in the "harder to do" sciences it is common to distinguish an experiment from a
quasi-experiment.
Part of the difficulty of these fields is that we can not (or ethically may
not) manipulate many independent variables. Therefore we lose the opportunity
to assert "et ceteris paribus" "everything else being equal" that is part of a
true experiment.
We cannot randomly assign people to ethnic categories. There can be a plethora
of plausible rival hypotheses. see
http://www.personnelselection.com/adverse.impact.htm
Alan McLean wrote:
> Some more comments on hypothesis testing:
>
> My impression of the �hypothesis test controversy�, which seems to exist
> primarily in the areas of psychology, education and the like (this is
> coming from someone who has been involved in education for all my
> working life, but with a scientific/mathematical background), is that it
> is at least partly a consequence of the sheer difficulty of carrying out
> quantitative research in those fields. A root of the problem seems to be
> definitional. I am referring here to the definition of the variables
> involved.
>
> In, say, an agricultural research problem it is usually easy enough to
> define the variables. For a very simple example, if one is interested in
> comparing two strains of a crop for yield, it is very easy to define the
> variable of interest. It is reasonably easy to design an experiment to
> vary fairly obvious factors and to carry out the experiment.
>
> In the �soft� sciences it is easy enough to identify a characteristic of
> interest � the problem is how to measure it. If I am interested in the
> relationship between ability in statistics and ethnic background, for
> example, I measure the statistics ability using a test of some sort; I
> measure ethnic background by defining a set of ethnicities. There are
> literally an infinite number of combinations that I can use � infinitely
> many different tests, all purporting to measure �statistics ability�
> (even if I change only one word in a test, I cannot be absolutely
> certain of its effect, so it is a different test!), and a very large
> number of definitions of �ethnicity�.
>
> This is of course not news to anyone reading this. But I am coming to my
> point. Suppose I carry out an �experiment� � I apply the test to a group
> of people of varying ethnicity, score them on the test and analyse the
> results, including a hypothesis test to decide if statistics ability is
> related to ethnicity. This test might be a simple ANOVA, or a
> Kruskal-Wallis or a chi square test, depending on how I score the test.
>
> As I said earlier, a hypothesis test only helps the user to decide which
> of two models is probably better. The point of the above paragraphs is
> this: the definition of the models being compared includes the
> definition of the variables used. If I reject the null model (a label I
> prefer to �null hypothesis�) � that is I decide that the alternative
> model is (likely to work) better � I am NOT saying that there is a
> relationship between statistics ability and ethnicity. All I am saying
> is that there is a relationship between the two variables I used.
>
> Please note that the test is not saying this � I am. The test merely
> gives me a measure of the strength of the evidence provided by the data
> (�significant at 1%� or �p-value of .0135�); this measure is only
> relevant if the models I have used are appropriate. I can use other
> evidence (experience is what we usually use! but there may be related
> tests that help) to decide if the model is appropriate.
>
> So there are three levels at which judgement is used to make decisions:
> deciding what variables are to be used to measure the characteristics
> of interest, and how any relationship between them relates to the
> characteristics
> deciding on the model to be used, and how to test it
> deciding the conclusion for the model
>
> In each of these there is evidence we use to help us make the decision.
> The hypothesis test itself provides the test for the third.
>
> Finally (at least for the moment) � whether we choose the null or
> alternative model, it IS a decision. In research, accepting the null
> means that we decide to accept it at least for the moment, so it is not
> necessarily a committed decision. On the other hand, if a line of
> investigation is not yielding results, the researcher is likely to not
> continue on that line � so it is a decision which does lead to an
> action.
>
> For non research applications such as in quality control, accepting the
> null model quite clearly is a decision to act on the basis of that. For
> example, with a bottle filling machine which is periodically tested as
> to the mean contents, the null is that the machine is filling the
> bottles correctly. Rejecting the null entails stopping the machine;
> accepting it means the machine will not be stopped.
>
> Traditional hypothesis testing does incorporate a decision-theoretic
> loss function � the p-value.
>
> Regards again,
> Alan
>
> --
> Alan McLean ([EMAIL PROTECTED])
> Department of Econometrics and Business Statistics
> Monash University, Caulfield Campus, Melbourne
> Tel: +61 03 9903 2102 Fax: +61 03 9903 2007
>
> ===========================================================================
> This list is open to everyone. Occasionally, less thoughtful
> people send inappropriate messages. Please DO NOT COMPLAIN TO
> THE POSTMASTER about these messages because the postmaster has no
> way of controlling them, and excessive complaints will result in
> termination of the list.
>
> For information about this list, including information about the
> problem of inappropriate messages and information about how to
> unsubscribe, please see the web page at
> http://jse.stat.ncsu.edu/
> ===========================================================================
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
