Rich Ulrich wrote:
>
> I have a question of my own - I think FAMILY is not, in general, the
> same as EXPERIMENT.
>From my class notes...
It is difficult to give a firm rule for when multiple
comparison procedures should be used. The most widely
respected statistician in the field was Rupert G. Miller, Jr.
who made no pretense of being able to resolve the issue but
offered some guidelines in his book Simultaneous Statistical
Inference, 2nd edition (Chapter 1, section 5, emphasis is
his): [The rest of this post is Miller speaking.]
Time has now run out. There is nowhere left for
the author to go but to discuss just what constitutes
a family [of comparisons to which multiple comparison
procedures are applied]. This is the hardest part of
the book because this is where statistics takes leave
of mathematics and must be guided by subjective
judgment. . . .
Provided the nonsimultaneous statistician [one who
never adjusts for multiple comparisons] and his client
are well aware of their error rates for groups of
statements, and feel the group rates are either
satisfactory or unimportant, the author has no quarrel
with them. Every man should get to pick his own error
rates. SImultaneous techniques certainly do not
apply, or should not be applied, to every problem.
[I]t is important to distinguish between two types
of experiments. The first is the preliminary, search-
type experiment concerned with uncovering leads that
can be pursued further to determine their relevance to
the problem. The second is the final, more definitive
experiment from which conclusions will be drawn and
reported. Most experiments will involve a little of
both, but it is conceptually convenient to being
basically distinct. The statistician does not have to
be as conservative for the first type as for the
second, but simultaneous techniques are still quite
useful for keeping the number of leads that must be
traced within reasonable bounds. In the latter type
multiple comparison techniques are very helpful in
avoiding public pronouncements of red herrings simply
because the investigation was very large.
The *natural family* for the author *in the
majority of instances* is the *individual experiment*
of a *single researcher*. . . . The loophole is of
course the clause *in the majority of instances*.
Whether or not this rule of thumb applies will depend
upon the size of the experiment. Large single
experiments cannot be treated as a whole without an
unjustifiable loss in sensitivity. . . . *There are
no hard-and-fast rules for where the family lines
should be drawn, and the statistician must rely on his
own judgment for the problem at hand*. (Chapter 1,
section 5)
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
