Re: p is continuous, not dichotomous

[G. Marc Turner]

I agree that the significance level is continuous, and yes, probabilities range from 0 to 1. But, the decision of whether something is statistically significant still seems like an either/or situation. I guess it depends on how you phase the question: Is the question "how likely is it?" Or is the question, "is this so unlikely that I don't think it was caused by chance and I'm going to say that it has to be below p=.05 to meet that criteria?" (not my favorite phrasing, but I'm still half asleep... need more coffee...)

As others have mentioned, if you set an alpha of .05 as the criteria for something to be statistically significant, stick to it. To me we're saying, "Well, I set alpha at .05. I didn't get that, but I did get .08. So, now I think that .08 is close enough to .05 to say there is something statistical significant (though only marginally so)... but it is still significant." To me, anything above your preset criteria is not significant if you said it had to be below that level to be significant. It isn't marginally significant, but rather marginally NON-significant. And, unless it is a moving target it is neither approaching nor avoiding significance...

Now for the bigger issue which seems to have us worked up: Is the p-value / statistical significance the only thing to consider? Of course not... which is why we're having fun talking about effect size, power, and all the joys of things that keep students up late at night. I offered up power & sample sample size as an example of things to think about, and Paul and others have elaborated on many of the other issues related to it.

But, in my mind, if you set a criteria for the test, you should stick to it for making that decision. So far, no one has said anything that convinces me that if you say it has to be below .05 to be statistically significant, that suddenly .08 is below .05. (And yes, I know that isn't the only thing to consider... but considering other things still won't make .08 below .05).

- Marc

At 11:45 PM 11/11/2002 -0500, you wrote:

Last I checked, the significance level, p, was a probability (the
conditional probability of obtaining results as more discrepant with the
null than are those in the current sample), and probabilities vary
CONTINUOUSLY from 0 to 1.  At least that is what Jack Cohen told me.

I suggest that we simply treat p as a measure of how well the data fit with
the null hypothesis.  P = .08 is very poor fit, p = .04 is not much poorer,
and p = .80 tells me that we got just about what we would expect were the
null true.

Karl W.

---
You are currently subscribed to tips as: [EMAIL PROTECTED]
To unsubscribe send a blank email to [EMAIL PROTECTED]