On Thu, 13 Apr 2000, Alan McLean wrote:
> Some more comments on hypothesis testing:
>
> My impression of the �hypothesis test controversy�, which seems to exist
> primarily in the areas of psychology, education and the like is that it
> is at least partly a consequence of the sheer difficulty of carrying out
> quantitative research in those fields. A root of the problem seems to be
> definitional. I am referring here to the definition of the variables
> involved.
>
> In, say, an agricultural research problem it is usually easy enough to
> define the variables. For a very simple example, if one is interested in
In addition to defining the variables some areas do a better job of
defining and therefore testing their models. The ag example is one where
not only the variables are relatively clear so are the models. That is
there is one highly plausible reason for rejecting a null that fertilizer
does not effect crop production: Fertilizer increases crop production.
You have rejected a model of no effect in favor of a model positing an
effect.
But in some areas in psychology you will have a situation where many
theoretical perspectives predict the same outcome relative to a zero
valued null while the zero valued null reflects no theoretical
perspective. In this situation rejecting a zero valued null supports all
theoretical perspectives equally and differentiates among none of them.
In a recent example a student was citing the research literature
supporting the convergent validity of some measure. The evidence used by
all investigators was that the null of rho = 0 was rejected. I've seen
this same thing many times, but this time I saw something different. The
smallest sample (n about 95) failed to reject rho = 0 while the remaining
samples (all n's > 200) successfully rejected rho = 0 and convergent
validity was declared. (No r's were actually reported in this review.)
A quick thought experiment, and check of critical value tables, suggests
that the best estimate of rho from the evidence provided is some value
greater than 0 but less than .20.
In this case it seems to me that testing the default zero valued null was
misleading rather than informative. In addition to convergent validity it
seems to me that correlations in the range 0 - .20 could easily be
explained by at least a couple of other competing models that would not
support the conclusions drawn. Only the most trivial link between
theoretical models and statistical hypotheses exist in this case.
Using Alan's ethnicity and statistical ability example, and assuming for
the moment that all measures were useful, the first time we reject a no
effect null we have some sort of useful information. Now, imagine that 12
researchers generate 12 different hypotheses explaining the cause of these
differences. Current practice has all 12 of these researchers collecting
data and testing to eliminate the chance model and then declaring that
their hypothesis has been confirmed.
I agree that measurement is a problem, but even with good measurement the
lack of connection between statistical hypotheses and theoretical
predictions is a fatal flaw in too many areas.
Michael
>
> Regards again,
> Alan
>
>
> --
> Alan McLean ([EMAIL PROTECTED])
> Department of Econometrics and Business Statistics
> Monash University, Caulfield Campus, Melbourne
> Tel: +61 03 9903 2102 Fax: +61 03 9903 2007
>
>
>
>
> ===========================================================================
> This list is open to everyone. Occasionally, less thoughtful
> people send inappropriate messages. Please DO NOT COMPLAIN TO
> THE POSTMASTER about these messages because the postmaster has no
> way of controlling them, and excessive complaints will result in
> termination of the list.
>
> For information about this list, including information about the
> problem of inappropriate messages and information about how to
> unsubscribe, please see the web page at
> http://jse.stat.ncsu.edu/
> ===========================================================================
>
*******************************************************************
Michael M. Granaas
Associate Professor [EMAIL PROTECTED]
Department of Psychology
University of South Dakota Phone: (605) 677-5295
Vermillion, SD 57069 FAX: (605) 677-6604
*******************************************************************
All views expressed are those of the author and do not necessarily
reflect those of the University of South Dakota, or the South
Dakota Board of Regents.
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
