Michael Granaas wrote (in part):
> And if the data are consistent with the null we do what? Act as if the
> null were true? Act as if nothing were true?
In general, that last is exactly what not rejecting the null permits.
Recall that failing to reject the null is (among other things) the almost
certain result of using an undersized sample. Therefore, any general
interpretation "if you do not reject the null then believe...." must cover
that eventuality. If you have separate rules for interpretation depending
on sample size you are getting back into interval estimation, a better place
to be anyway.
> Every research project should plausibly add to our knowledge base. But,
> if the null is a priori false failure to reject is just that a failure and
> waste of time.
Exactly. And most nulls *are* a priori false.
> > > Isn't that what we do in our experiments all the time? We assume that
our
> > > experimental manipulation has no effect, which is plausibly true at
least
> > > for some time, and then we try to disprove that estimate of the
effect.
> > > Failing to do so we act as if the effect were absent (or so small as
to be
> > > absent for all practical purposes).
> >
> > We have no right to do the latter unless we ahve actually estimated
> > effect and it *is* that small.
>
> Depends on the context: applied or basic research. In basic research I
> would agree that no effect should be ignored no matter how small. But,
> for the applied person some effects are so small that they should be
> ignored. Gender differences in academic abilities are likely to be
> interesting to researchers trying to understand gender differences. But,
> they are so small that using their existance to provide career advice
> is useless at best and terribly harmful at worst.
I think you may have missed my point, which was that if we conclude that
an effect is too small to be of interest, it should be because it is too
small to be of interest, not because it is too small to detect with the data
easily available. (And the fox, having tried to reach the grapes and failed,
said "Aaaaah! Those grapes were not statistically significant anyway.)
> > And again:
> >
> > > > (7) Back at the beginning we wanted a yes-or-no answer. Henced
fixed
> > > > alpha testing and the pretence that we "accept" null hypotheses.
> > >
> > > If the null is plausibly true we need no pretense. We accept the null
as
> > > true until something better comes along. I personally have accepted
the
> > > notion that psi powers do not exist despite the fact that all I have
is a
> > > string of failures to reject the null as evidence.
> >
> > Spoken like a Bayesian, sir!
>
> Hmm, I've never been called a Bayesian before. I tend to think of myself
> as a frequentist with Popperian tendencies.
>
> > But if you talk the talk you should also
> > walk the walk. Hypothesis testing does not give you any way to formally
> > introduce the idea that a null is "plausibly true".
>
> In our best understanding of Fisher's work this would indeed seem to be
> what he said. In the world of 1928 his notion of null hypotheses was
> certainly a great leap forward. But just because Fisher said that there
> was no way to introduce the idea that the null was plausibly true does not
> make it so. Fisher also said that there was no such thing as power or
> type II errors or alternative hypotheses. These are perhaps not
> universally accepted concepts, but they certainly have a wide acceptance.
>
> Unfortunately in hypothesis testing where the null may not be plausibly
> true we have no means of discovering truth, only falsehood. We conclude
> that either the null is false and something else must be true or we
> conclude that the evidence is inconclusive and we know nothing. In the
> first case we know that something is true but we do not know what that
> something is. In the second we know nothing.
Whatever Fisher said - and I am not an historian of statistics -
that last paragraph does indeed seem to sum up the situation; and it would
seem like a good argument against hypothesis testing in many situations.
>If I am to know what is true I must first identify all of the things that
>may plausibly be true and then systematically eliminate them until I find
>one that I cannot eliminate. That one then must be true.
*If* you can eliminate all but one, it would follow that the one was
true.
In practice, this is rarely if ever the case.
In order to
>achieve this I must first accept the plausibility of the hypothesis that I
>am testing and make an effort to reject it. Failure to reject means that
>it stays on the list of plausible truths, for now. The more failures to
>reject the greater my confidence in the plausibility of the hypothesis.
No! (or, at least, *my* confidence would not necessarily increase.)
Suppose that the mean IQ of a group is in fact 100 with standard deviation
15; and I, using sample sizes of 10 repeatedly, try to test the hypothesis
that the mean is 102. I will do this 1000 times (MINITAB simulation:
MTB > random 1000 c1-c10;
SUBC> normal 100 15.
MTB > rmean c1-c10 c15
MTB > rstd c1-c10 c16
MTB > let c17 = abs(c15-102) < c16*2.262/sqrt(10)
MTB > mean c17
mean of c17 = 0.932
and we see that 93% of my experiments fail to reject H0. Should I now
believe it to be true?
The problem is that failure to reject means *either* that the null is
true *or* that the sample size too small *or* both; and a classical
hypothesis test on its own does not tell you which.
-Robert Dawson
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
