Rich,
Both Radford Neal and I have asked
for a statistical rationale supporting
your claim that a significance test
that you advocated
can provide useful information when applied
to the MIT senior biologist data. You
haven't provided one. Instead, you
cite from a web statistics guide which
in turn provides no rationale.
It is now quite apparent that you
have no rationale, only prejudices.
This may be acceptable to the people
who come to you for consulting, but
this is a different forum, with different
standards.
Further comments are interspersed:
On Thu, 22 Feb 2001 18:21:41 -0500, Rich Ulrich <[EMAIL PROTECTED]>
wrote:
>On Mon, 19 Feb 2001 04:27:24 GMT, [EMAIL PROTECTED] (Irving
>Scheffe) wrote:
>
>> In responding to Rich, I'll intersperse selected comments with
>> selected portions of his text and append his entire post below.
>
> - I'm not done with the topic yet. But it is difficult to go on from
>this point.
>
>I think the difficulty is that JS has constructed his straw-man
>argument about how "hypotheses" are handled; and since it
>is a stupid strategy, it is easy for him to claim that it is fatally
>flawed.
All the referents are unclear. I didn't construct any straw man
arguments, and you haven't made clear what you are talking about.
You are the one who examined nonrandom data, representing citation
counts over a 12 year period for senior male and female MIT biologists
matched for year of Ph.D. You look at these data, which
show a HUGE difference in performance between the men and women,
and declare that a significance test is necessary. But you
cannot provide any mathematical justification for the test.
I gave several examples to try to jar you into realizing that
a statistical test on the data cannot answer the question you
want answered.
>
>From his insistence on his "examples," it seems to me that he
>believes that someone else is committed to using p-levels in a strict
>way, by beating 5%.
Not so. If you were following the logic of the many examples I've
presented, you could see that you can construct a reductio ad
absurdem for any of the types of significance tests you are
proposing. If I believed strictly in hypothesis testing
with a 5% significance level, I doubt that I'd have written
an extensive article advocating confidence interval replacements
for many of the classic hypothesis tests employed in the social
sciences, and giving the precise, exact procedures for
constructing these confidence intervals.
>That's certainly not the case for me, and I
>doubt if anyone defends or promotes it, outside of carefully designed
>Controlled Random Experiments.
>
It is not the case for me, either, and so everything that follows is
irrelevant.
>Despite the fact that I could not make sense of WHY he wanted
>his example, it turns out -- after he explains it more -- that my own
>analysis covered the relevant bases. I agree, if you don't have
>"statistical power," then you don't ask for a 5% test, or (maybe)
>any test at all. The JUSTIFICATION for having a test on the MIT
>data is that the power is sufficient to say something.
In order to talk meaningfully about "power", you have to have
a statistical rationale. As I have repeated numerous times,
you have no statistical rationale. You simply "feel like"
you "should" compute a statistical test, when all the assumptions
on which the procedure is based are violated in the data you
are applying the procedure to.
Power to detect what? Under what distributional assumptions?
>
>And what it said is that Jim did BAD INFERENCE. I said that a
>couple of times. I regret that I may have confused people with
>unnecessary words about "inference."
> Outlier => No central tendency => Mean is BAD statistic;
>careful reader insists on more or better information before asserting
>there's a difference.
What "outlier" are you referring to? What statistical rule did you
use to determine the "outlier"?
The MIT paper included all the raw data. At no point did I or my
coauthor state that we were doing inference on means. (Actually,
a 2 sample t-test done on these data is significant at the .05
level, but we never imagined computing one.)
Here are the raw data for the citation counts for the 5 senior
MIT female biologists and 6 males who graduated from 1970-76.
Males Females
-----------------------
12830 2719
11313 1690
10628 1301
4396 1051
2133 935
893
-----------------------
These data are based on 12 years worth of records, from 1989-2000.
The above could be broken down in numerous other ways. For example, we
could produce citation counts per year, try to perform some kind of
correction for the highly specific areas the individuals publish in,
etc. Time series could be examined.
However, these data are anything but a random sample. MIT is one of
the most selective universities in the world in terms of whom it
hires.
>
>I asserted that more than once.
>
>Optimistically, my own data analysis technique might be described as
>"starting out with everything Jim might figure out and conclude from
>the data, and adding to that, flexible comparisons from related
>fields, and other statistical tools." -- It was quite annoying for
>me to read where he implicitly says, "You, idiot, would HAVE to
>decide otherwise." I mean, I thought I wrote a lot clearer than that.
I challenge you to bring the above paragraph to any high school
English teacher and ask him/her to evaluate it for clarity.
Since I freely admit it is possible to do various kinds of inferences
on the above data, I don't think your quote from Garson is
particularly relevant.
But let us now *return* to those comments you made originally, that
you thought were so clear. You began by saying
>- and if you want to know something about how unlikely it was to
>get means that extreme, you can randomize. Do the test.
You then went on to say
> can't say that I have absorbed everything that has been argued.
>But as of now, I think Gene has the better of it. To me, it is not
>very appropriate to be highly impressed at the mean-differences,
>when TESTS that are attempted can't show anything. The samples
>are small-ish, but the means must be wrecked a bit by outliers.
My assertion is that the 6 males have had substantially
more scientific impact than the 5 females. [And I'm sure
that any knowledgeable biologist will agree.] The highest
ranking female has fewer citations than the 4th ranking
male, and half the males have about 4 times as many citations
as the top ranking female. Randomization tests, besides
having no statistical basis in this situation, throw
away metric information in the citation data.
Remembering that these data were gathered over a 12 year span,
and that they are designed ONLY to answer the question, "Is
it possible that the higher departmental perks and status
of these males is due to performance," I think your commentary
was ludicrous, Rich.
The MIT fiasco is an important social issue, because the
"MIT Report" is being used as a template to support radical
affirmative action policies that are going to impinge negatively
on hundreds of young male scientists over the next few years.
So, again, Rich, I'll ask you:
a. What, precisely, is your statistical rationale for performing
a randomization test on the above data?
b. Name the quantities you are estimating with the test. Tell how and
why you think the test, and its p-level, assess that quantity.
Please be specific. I want to know what you think, not what some
obscure writer from 1986 thought might be a good idea, based
on some unknown rationale that he cannot describe coherently.
c. How did you determine that I had adequate "power" to do
your test, when you have not yet told me what distribution you
think the test has?
I'm very confident that you cannot provide a rationale,
because you have none.
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
