It is a great book, and has one of my favorite phrases, Abelson¹s First Law of statistics: ³Chance is lumpy.² By which he is saying this: in any collection of actually random data, it is not necessarily interesting that you find a pattern here or there, a lump of data that goes in a favored or disfavored direction. Lumps happen, but are not necessarily meaningful. They may be type one or type two errors if, when you dipped your ladle into the pool of data, you happened to catch a lump rather than the more apparently random part.
-- Paul Bernhardt Frostburg State University Frostburg, MD, USA On 1/9/09 10:32 PM, "Claudia Stanny" <[email protected]> wrote: > > > > > > > > Abelson wrote a book called Statistics as principled argument (1995) that > includes a great discussion of the rationale for post hoc tests and other > unanticipated and unplanned analyses of a data set. > > The argument he presents is that the researcher¹s job is to figure out what > story the data have to tell and report the analyses that tell that story. If > you did not have the insight to frame a particular question during the > planning stage of your research but realize that the data set could be used to > answer a meaningful question if some revised analysis is used, there is no > good reason to claim that the new analysis may not or should not be conducted. > While this might sound like encouragement to fish the data set for whatever > significant finding you can produce with a new analysis, I do not think this > is what Abelson intends. This is suggested by his argument that one > statistically significant finding (especially following an extensive fishing > expedition) does not make an effect any more than ³one swallow makes a > summer². > > If a researcher can make a case that messiness and ambiguity in the data set > could be cleaned up by adding a covariate that hadn¹t been considered at the > outset, the researcher would be a little foolish to not do the new, more > clear-cut analysis. Sometimes we realize that we can answer an additional > question by partitioning the groups in a slightly different way or using some > characteristic of the participants, experimental procedures, or other > variables that we can define after the fact to produce a new analysis with > additional variables. > > The rigidity of detailing the plan for data analysis in advance and forbidding > any other approaches strikes me as a little silly. It seems to encourage > people to think about data analysis as a magical process that reveals ³truth² > the way that taking one¹s goat or chicken to the oracle and asking for a > reading of the entrails was once thought to forecast the future. > > Abelson¹s book makes a great case for exploring the data set until you > understand what it has to say. The statistical arguments (and the strength of > these arguments) is based on the underlying rationale for the analyses done. > If an unplanned set of comparisons will do a good job of answering a new > question we didn¹t think of at first but thought of as we puzzled over the > data, this rationale for the new analysis might give us a strong argument. If > the rationale for the unplanned analysis is difficult to justify except that > it only analysis we could discover that produced the magic p value of < .05, > the argument is weak and the findings are unlikely to stand up to replication. > > If you haven¹t read this little book, I strongly recommend it. It is a real > gem, full of good sense about the use of statistics as a decision tool. > > Claudia J. Stanny, Ph.D. > Director, Center for University Teaching, Learning, and Assessment > Associate Professor, Psychology > University of West Florida > Pensacola, FL 32514 5751 > > Phone: (850) 857-6355 or 473-7435 > e-mail: [email protected] > > > --- > To make changes to your subscription contact: > > Bill Southerly ([email protected]) > --- To make changes to your subscription contact: Bill Southerly ([email protected])
