In article <[EMAIL PROTECTED]>,
> Chris: That's not what Jerry means. What he's saying is that if
> your sample size is large enough, a difference may be statistically
> significant (a term which has a very precise meaning, especially to
> the Apostles of the Holy 5%) but not large enough to be practically
> important. [A hypothetical very large sample might show, let us say,
> that a very expensive diet supplement reduced one's chances of a heart
> attack by 1/10 of 1%.]
Firstly, I think we can thank publication pressures for the church of
the Holy 5%. I go with Keppel's approach in suspending judgement for mid
range significance levels (although we should do this for nonsignificant
results anyway as they are inherently indeterminant).
Wrt to your example, it seems that the decision you are making about
practical importance is purely subjective. In any number of alternative
situations a .01% effect could have major implications, practical and
theoretical.I regard this as less a fundamental flaw with hypothesis
testing and more a question of expermental design and asking the right
questions to begin with.
>Alternatively, in an imperfectly-controlled
> study, it may show an effect that - whether large enough to be of
> interest or not - is too small to ascribe a cause to. [A moderately
> large study might show that some ethnic group has a 1% higher rate of
> heart attacks, with amargin of error of +- .2% . But we might have, or
> an effect of this size, no way of telling whether it's due to genes,
> diet, socioeconomic factors, recreational drugs, or whatever.]
Surely the ambiguity of this outcome is the result of the lack of
experimental control. If the effects of genetics, diet etc. are not
appropriately controlled, it doesn't matter what sample size is
used - the outcome will be always be equivocal. What it does suggest is
that, irrespective of sample size, we must be vigilant in controlling
for extraneous variables. Is it fair to consider this a flaw of
hypothesis testing? We can hardly blame the tools for not working
properly if they are not used correctly.
Chris
Sent via Deja.com http://www.deja.com/
Before you buy.
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
