I would appreciate feedback on the following from list members.
I recently participated in a discussion at a conference that revolved around
effect sizes. The discussion had to do with the clinical value of a set of
predictors based on field studies. In these studies, the predictors (which
were all dichotomous, positive-negative judgments based on a clinical test)
were related to a series of criteria, some of which were dichotomous and
some of which were quantitative. Pearson correlations were computed for all
comparisons, and d statistics were also generated for all comparisons
involving quantitative criteria. An important point to keep in mind was
that the base rates for the predictors (and many of the criteria) were quite
skewed; in general, only about 1 in 15 members of the sample were positive
on any one of the predictors. These were field studies, so the skew
presumably represents real-world base rates.
The basis for the discussion was the extreme difference in conclusions one
would draw based on whether you computed correlations or d. Because of the
skew, a d value of .71 (generally considered a "large" effect) translated
into an r value of .15. A d value of .31 (medium-sized) transformed to an r
value of .07.
The discussion that followed focused on which is the "better" effect size
for understanding the usefulness of these predictors. Some of the key
points raised:
1. r is more useful here for several reasons:
a. It is generally applicable to both the dichotomous and quantitative
criteria.
b. The concept of "proportion of overlapping variance" has more general
usefulness than "mean difference as a proportion of standard deviation."
c. The results of the correlational analyses were more consistent with
the results of significance tests, that is, even with large samples (N >
1000), many of the expected relationships proved to be nonsignificant.
2. d is more useful precisely because it is relatively resistant to the
impact of skew, unless group SDs are markedly different.
3. A third, less important issue, was raised in response to point 2. If
effect size measures that are resistant to skew are more desirable, is there
one that could be applied to both dichotomous and quantitative criteria? If
not, which would be the "better" effect size measure for dichotomous
criteria:
a. the tetrachoric r: one person recommended this on the grounds that it
is conceptually similar to the Pearson r and therefore more familiar to
researchers.
b. the odds ratio: recommended because it does not require the
distributional assumptions of the tetrachoric r.
The key issue on which I'd like your input, although please feel free to
comment on any aspect, is this. Given there is real-world skew in the
occurrence of positives, does r or d present a more accurate picture?
Should we think of these as small or medium-to-large effect sizes?
-----
Robert McGrath, Ph.D.
School of Psychology T110A
Fairleigh Dickinson University, Teaneck NJ 07666
voice: 201-692-2445 fax: 201-692-2304
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
