I have a process that attempts to calculate the statistical significance
of the result it produces. If I run this process with 100 independent
inputs, I generate 100 probabilities that the result occurred due to
chance. I can then plot a qqplot of 1:100/100 vs. sort(probabilities),
and if the points fall on a straight line, I can say that my process's
estimation of signifance for each result is fairly accurate (as long as
I've fed the process random data as input in the first place, which I
have), i.e. only 1 out of a 100 times did I assess the signifance as
low as 0.01 or worse, and only 10 out of 100 times did I assess the
signifance as 0.1 or worse ...
Is there some way of capturing the meaning of this qqplot numerically? I
actually have many different "batches" of independent inputs (each batch
represents independent inputs which share a certain quality) and many
different ways of running the process, and I'd like to investigate how
well (or not well) my signifance estimates are doing across these
variables without examining (or publishing) hundreds of qqplots.
Thanks,
-Aaron
P.S. I apologize for the nonspecifics of my post - if it truly matters,
I'd be happy to provide more detail.
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
