most software will compute p values (say for a typical two sample t test of
means) by taking the obtained t test statistic ... making it both + and -
... finding the two end tail areas in the relevant t distribution ... and
report that as p
for example ... what if we have output like:
N Mean StDev SE Mean
exp 20 30.80 5.20 1.2
cont 20 27.84 3.95 0.88
Difference = mu exp - mu cont
Estimate for difference: 2.95
95% CI for difference: (-0.01, 5.92)
T-Test of difference = 0 (vs not =): T-Value = 2.02 P-Value = 0.051 DF = 35
for 35 df ... minitab finds the areas beyond -2.20 and + 2.02 ... adds them
together .. and this value in the present case is .051
now, traditionally, we would retain the null with this p value ... and, we
generally say that the p value means ... this is the probability of
obtaining a result (like we got) IF the null were true
but, the result WE got was finding a mean difference in FAVOR of the exp
group ...
however, the p value does NOT mean that the probability of finding a
difference IN FAVOR of the exp group ... if the null were true ... is .051
... right? since the p value has been calculated based on BOTH ends of the
t distribution ... it includes both extremes where the exp is better than
the control ... AND where the cont is better than the exp
thus, would it be fair to say that ... it is NOT correct to say that the p
value (as traditionally calculated) represents the probability of finding a
result LIKE WE FOUND ... if the null were true? that p would be 1/2 of
what is calculated
this brings up another point ... in the above case ... typically we would
retain the null ... but, the p of finding the result LIKE WE DID ... if the
null were true ... is only 1/2 of .051 ... less than the alpha of .05 that
we have used
thus ... what alpha are we really using when we do this?
this is just a query about my continuing concern of what useful information
p values give us ... and, if the p value provides NO (given the results we
see) information as to the direction of the effect ... then, again ... all
it suggests to us (as p gets smaller) is that the null is more likely not
to be true ...
given that it might not be true in either direction from the null ... how
is this really helping us when we are interested in the "treatment" effect?
[given that we have the direction of the results AND the p value ...
nothing else]
==============================================================
dennis roberts, penn state university
educational psychology, 8148632401
http://roberts.ed.psu.edu/users/droberts/drober~1.htm
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
