On Tue, 11 Apr 2000, Robert Dawson wrote:
>
> ----- Original Message -----
>
> I wrote:
> > > (4) In order to avoid circular logic, we *cannot* assume what we
> want to
> > > prove, in order to compute the probability. We can however assume it for
> a
> > > contradiction. Therefore...
>
> and Michael Granaas responded
> > This (point 4) is certainly what we have been lead to believe, but I
> > question the assumption. Do we not in fact teach that we are to act as if
> > the null is true until we can demonstrate otherwise?
>
> I certainly don't. We *compute* as if the null was true, whether we
> believe it or not; then we either conclude that (null + data) is implausible
> or that the data are consistent with the null.
And if the data are consistent with the null we do what? Act as if the
null were true? Act as if nothing were true? In a pure interpretation of
this approach we must act as if there were no knowledge (null not
rejected) or only very weak knowledge (effect is in the ________
direction). The first is a complete waste of effort and the second
provides only the weakest bit of sketchy knowledge.
Every research project should plausibly add to our knowledge base. But,
if the null is a priori false failure to reject is just that a failure and
waste of time.
>
> > Isn't that what we do in our experiments all the time? We assume that our
> > experimental manipulation has no effect, which is plausibly true at least
> > for some time, and then we try to disprove that estimate of the effect.
> > Failing to do so we act as if the effect were absent (or so small as to be
> > absent for all practical purposes).
>
> We have no right to do the latter unless we ahve actually estimated
> effect and it *is* that small.
Depends on the context: applied or basic research. In basic research I
would agree that no effect should be ignored no matter how small. But,
for the applied person some effects are so small that they should be
ignored. Gender differences in academic abilities are likely to be
interesting to researchers trying to understand gender differences. But,
they are so small that using their existance to provide career advice
is useless at best and terribly harmful at worst.
>
> And again:
>
> > > (7) Back at the beginning we wanted a yes-or-no answer. Henced fixed
> > > alpha testing and the pretence that we "accept" null hypotheses.
> >
> > If the null is plausibly true we need no pretense. We accept the null as
> > true until something better comes along. I personally have accepted the
> > notion that psi powers do not exist despite the fact that all I have is a
> > string of failures to reject the null as evidence.
>
> Spoken like a Bayesian, sir!
Hmm, I've never been called a Bayesian before. I tend to think of myself
as a frequentist with Popperian tendencies.
> But if you talk the talk you should also
> walk the walk. Hypothesis testing does not give you any way to formally
> introduce the idea that a null is "plausibly true".
In our best understanding of Fisher's work this would indeed seem to be
what he said. In the world of 1928 his notion of null hypotheses was
certainly a great leap forward. But just because Fisher said that there
was no way to introduce the idea that the null was plausibly true does not
make it so. Fisher also said that there was no such thing as power or
type II errors or alternative hypotheses. These are perhaps not
universally accepted concepts, but they certainly have a wide acceptance.
Unfortunately in hypothesis testing where the null may not be plausibly
true we have no means of discovering truth, only falsehood. We conclude
that either the null is false and something else must be true or we
conclude that the evidence is inconclusive and we know nothing. In the
first case we know that something is true but we do not know what that
something is. In the second we know nothing.
Or perhaps you are taking an extremely traditional approach to this whole
topic just to tease me a bit. It matters not, these particular brain
cells have been in need of some exercise and I appreciate the work out
whether your challenges are genuine or just an excuse to play.
> Bayesian inference does.
>
>
> > > OK, it's a horrible kludge at best, and an evil ritual at worst; but
> > > *if* you start with those assumptions & goals, there's what comes out.
> >
> > Yes. Since I refuse to start with assumption that the null must be false
> > to be useful I end up in a somewhat different place.
>
> Starting from there, you *should* end up in Bayesian inference, or at
> least likelihood-based inference. If you want to be able to conclude that
> the null is plausibly true, you must not start out by assuming its truth
> even
> as a working assumption. You can assume nothing, and compare likelihoods;
> or
> you can assume some _probability_ for the null and see what the data do
> to that assumption, via Bayes' theorem. But if you assume truth, you can
> get
> only a contradiction (null + data => improbable) or a tautology
> (null + data => null) that does not function as a proof.
Unfortunately rejecting a priori false hypotheses proves nothing worth
knowing. The notion that we can only reject hypotheses and never prove
them really does not allow us to ever know much of anything beyond the
fact that some finite number of things (values, effects) are
false/non-existant.
The very act of hypothesis testing forces us to assume that there is good
reason to conduct the test. We have to know something. Fisher seems to
have claimed that by repeatedly rejecting the no effect hypothesis we have
shown that we know that some effect does in fact exist. That is he
seems to have assumed that we need to know what is true to state what
is false.
Unfortunately this is the weakest possible proof of truth. I also know
that your birthday is not february 30th. I know that you were not born in
the year 12 b.c. The fact that I know these things to be false does not
in any way prove that I know anything beyond the nature of the dominant
calender system in western society and that I understant that people do
not live for 2000 years. I have not proved that I know anything about
you.
If I am to know what is true I must first identify all of the things that
may plausibly be true and then systematically eliminate them until I find
one that I cannot eliminate. That one then must be true. In order to
achieve this I must first accept the plausibility of the hypothesis that I
am testing and make an effort to reject it. Failure to reject means that
it stays on the list of plausible truths, for now. The more failures to
reject the greater my confidence in the plausibility of the hypothesis.
This approach allows me to ask questions about the nature of effects that
are absent from the classic "all nulls are false" approach. Having
identified an effect I can ask questions about its specific nature or
size. I can ask if the effect size is reasonably estimated as theta-hat.
More importantly I can test that estimate by stating it as a null and
trying to reject it. No matter how many times I reject 0 I cannot know
theta. Only by eliminating competing plausible estimates will I be able
to itentify the true magnitude of theta.
That is, Fisher advocated predicting what would not happen in order to
demonstrate that you knew what would happen. I advocate predicting what
will happen to demonstrate that you know what will happen. Predicting
what will happen is riskier because results could show that you are wrong.
Predicting what will not happen is risk free, either you were correct or
the results were inconclusive.
It is important that in testing our hypothesis of what may be true we MUST
make a good faith effort to reject it, otherwise we have proved nothing.
That is once we have assumed something and offered it as our null we must
become Fisherian again and try very hard to discredit that null. Only
then can we claim failure to reject as support for the null.
>
> -Robert Dawson
>
Michael
>
>
*******************************************************************
Michael M. Granaas
Associate Professor [EMAIL PROTECTED]
Department of Psychology
University of South Dakota Phone: (605) 677-5295
Vermillion, SD 57069 FAX: (605) 677-6604
*******************************************************************
All views expressed are those of the author and do not necessarily
reflect those of the University of South Dakota, or the South
Dakota Board of Regents.
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
