On 9 Jan 2009 at 10:53, Mike Palij wrote: > On Thu, 08 Jan 2009 20:04:23 -0800, Karl Wuensch wrote: > > I'm even less conservative than Stephen. I would not apply the > >Bonferroni adjustment. After all, these are PLANNED comparisons, eh? > > This is a curious point: > > Why should the state of knowledge (i.e., able to predict the size > of difference, the direction of a difference, etc.) affect the probability > of making an error of inference?
I've come to the opinion (based on reading and struggling with this issue over the years) that neither requiring a significant overall ANOVA nor prediction are critical. What is critical is how many questions you ask (number of comparisons) and why you want to do them (justification) Unlike the planned/post hoc concern, justification is open to inspection and evaluation by others. If you have only a few comparisons, and those tests can be justified as meaningful and essential to your investigation, then you can dispense with a Bonferroni or other correction for multiple comparisons, as Karl recommends above. However, as the number of tests goes up and especially if the justification for doing them is weak, the need for a Bonferroni correction correspondingly increases. This is admittedly subjective, but not excessively. Whether the data has been collected or not doesn't matter, as long as you don't let yourself be influenced by it (and others can see whether you have). You start by listing the set of meaningful and important comparisons which you must test, keeping the number to a minimum because more tests mean a more severe correction. If none are significant uncorrected, or if all are significant corrected for multiple comparisons, report that. For results which fall in-between (significant uncorrected, but not significant when corrected), report both results, and be correspondingly cautious in what you claim. Your reader can then decide for him/herself. This is essentially a planned approach, which I find far preferable to the clutter and confusion of testing with wild abandon, but it still requires correction for multiple comparisons in most cases. > > Consider t-tests, if we "know" that a difference is in a particular > direction, then using a one-tailed test will give us a lower critical > value and possibly make it easier to reject the null hypothesis that > the two means estimate the same population mean. But if our > "knowledge" is faulty or incomplete and the difference is in the > opposite direction we either (a) ignore a statistically significant > result because it was not in "right" direction, (b) say "Ooops!", > and claim we "planned" on doing a two-tailed test all along. > <snip> > It was for reasons > like this that Jack Cohen used to say that one-tailed tests should > not be done (how could one distinguish delusional self-confidence > from solid knowledge?) Absolutely. This isssue drives me nuts. One-tailed tests should be banned (almost), and certainly raise my suspicion whenever I see them used. But the issue isn't one of knowledge, and just because you make a prediction doesn't give you the right to use them. You must be able to swear (on the Bible, the Koran, or _The Design of Experiments_) that even if, contrary to all that you hold reasonable, the result falls in the "wrong" tail, you would have not the slightest interest in such an outcome, which you would declare to be totally meaningless. It's not easy to find such examples which fit this criterion allowing a one-tailed test. The only one I can recall is, say, that you're testing whether a therapy is better than a placebo. You predict that it is but, more important, you have absolutely no interest in an outcome which shows the therapy causes harm. Because harm and merely useless lead to the same conclusion: don't use the therapy. Even this isn't a good example, because a finding that a therapy causes deterioration could well have important implications. But a clinician might not care. Stephen ----------------------------------------------------------------- Stephen L. Black, Ph.D. Professor of Psychology, Emeritus Bishop's University e-mail: [email protected] 2600 College St. Sherbrooke QC J1M 1Z7 Canada Subscribe to discussion list (TIPS) for the teaching of psychology at http://flightline.highline.edu/sfrantz/tips/ ----------------------------------------------------------------------- --- To make changes to your subscription contact: Bill Southerly ([email protected])
