Abelson wrote a book called Statistics as principled argument (1995) that includes a great discussion of the rationale for post hoc tests and other unanticipated and unplanned analyses of a data set.
The argument he presents is that the researcher's job is to figure out what story the data have to tell and report the analyses that tell that story. If you did not have the insight to frame a particular question during the planning stage of your research but realize that the data set could be used to answer a meaningful question if some revised analysis is used, there is no good reason to claim that the new analysis may not or should not be conducted. While this might sound like encouragement to fish the data set for whatever significant finding you can produce with a new analysis, I do not think this is what Abelson intends. This is suggested by his argument that one statistically significant finding (especially following an extensive fishing expedition) does not make an effect any more than "one swallow makes a summer". If a researcher can make a case that messiness and ambiguity in the data set could be cleaned up by adding a covariate that hadn't been considered at the outset, the researcher would be a little foolish to not do the new, more clear-cut analysis. Sometimes we realize that we can answer an additional question by partitioning the groups in a slightly different way or using some characteristic of the participants, experimental procedures, or other variables that we can define after the fact to produce a new analysis with additional variables. The rigidity of detailing the plan for data analysis in advance and forbidding any other approaches strikes me as a little silly. It seems to encourage people to think about data analysis as a magical process that reveals "truth" the way that taking one's goat or chicken to the oracle and asking for a reading of the entrails was once thought to forecast the future. Abelson's book makes a great case for exploring the data set until you understand what it has to say. The statistical arguments (and the strength of these arguments) is based on the underlying rationale for the analyses done. If an unplanned set of comparisons will do a good job of answering a new question we didn't think of at first but thought of as we puzzled over the data, this rationale for the new analysis might give us a strong argument. If the rationale for the unplanned analysis is difficult to justify except that it only analysis we could discover that produced the magic p value of < .05, the argument is weak and the findings are unlikely to stand up to replication. If you haven't read this little book, I strongly recommend it. It is a real gem, full of good sense about the use of statistics as a decision tool. Claudia J. Stanny, Ph.D. Director, Center for University Teaching, Learning, and Assessment Associate Professor, Psychology University of West Florida Pensacola, FL 32514 - 5751 Phone: (850) 857-6355 or 473-7435 e-mail: [email protected] --- To make changes to your subscription contact: Bill Southerly ([email protected])
