Mike Palij wrote:
On Thu, 29 May 2008 12:50:40 -0700, Ken Steele writes:
A review article of these issues has appeared in timely fashion-
Sackett, P. R., Borneman, M. J., & Connelly, B. J. (2008).
High-stakes testing in higher education and employment: Appraising
the evidence for validity and fairness. American Psychologist, 63 (4),
215-227.
But I'd like to raise a larger issue and get feedback on the it.
One of the key problems in analyzing the relationship between
the SAT and measures of academic performance, such as GPA
at the end of the first year, is that of range restriction: colleges
usually do not admit people with SAT scores belows a certain
value so the range of values for both the SAT and GPA are both
restricted. However, consider the following two situations:
I agree with Mike that this is an issue that is poorly
understood. I just created a data set for a colleague to
simulate this effect. The set was a simulation of the
relationship between SAT and HS GPA. The original set began with
N = 100 and r = 0.8. I started reducing the set size based on
hypothetical college admission admission cutoffs. By the last
slice (OldIvy standards, SAT > 1399, N = 6), r = 0.19.
(2) Imagine that we have two variables, assume that both
might be normally distributed and that we can calculate a
correlation between the two variables. The problem is
variable X (say SAT scores) has a peculiar relationship to
variable Y (say GPA scores) namely
Prob (Y | X greater than a critical value) > 0 and
Prob (Y }X less than a critical value) = 0.
For example, a college only admits students with a combined
SAT Quantitative and SAT Verbal score of 1200. So, it is
only possible for have GPA grades for combined SAT >= 1200
but impossible to have a GPAs for combined SAT < 1200.
Theoretically, the combined SAT distribution is a truncated
normal distribution which has mathematical properties different
from ordinary normal distributions. The nature of this
situation is such that, becuse it is impossible to get GPAs
for combined SATs below 1200, the proper correlation is
between combined SAT >= 1200 and GPA. Any
correction for range restrictions refers to a "fantasy" situation
which cannott exist. Correlations based on corrections for
range restriction, therefore, are fantasy correlations because
they require assumptions that are false in reality (i.e., if the
college accepted students with SATs below 1200, what
would their GPAs be? But these people and values don't
exist -- indeed, would the college remain unchanged if it
admitted students with combined SATs < 1200?).
This comment is based on my reading of Rosenthal & Rosnow's
"Essentials" (2008, p. 349-353). Here is how they explain
Pearson's correction procedure for the change in r using the
example of some hypothetical academic skills test and the effect
on r going from HS GPA to the restricted group of 1st year
college GPA.
The essence of the correction/modification is that you need
variability measures from both the full group and the
restricted-range group. One r is converted to the other r by
weighting the change in variability.
I would presume that ETS has access to data such that it has a
good estimate of variabilities of SAT scores for HS students.
So the Sackett et al. correction does not seem as spooky as it
seemed on my first read.
I'd just mention that this thought isn't original with me though
I don't know whether others have applied it to the issue of
range-corrections to correlation. I don't have a copy of
Snedecor & Cochran's Statistical Methods handy but I do
recall an argument that was made in the context of the
analysis of covariance. Using covariates to control for various
variables in an ANOVA of an experiment might be problematic
if the removal of such effects is impossible in real life. The
ANCOVA may produce statistically significantly results but
the results may not generalize to the real world. If any one
has a copy of S&C, could they check on this, just in case
I'm dealing with a false memory or temporary psychotic break
with reality. ;-)
I happened to have a copy of S&C. I think the example you are
thinking of is on p. 431-432 which looks at an ANCOVA involving
school expenditures in 5 states adjusted for per capita income.
Their point is that if the per capita income were increased in
the poorer states then that increase would not necessarily be
apportioned to education.
Ken
---------------------------------------------------------------
Kenneth M. Steele, Ph.D. [EMAIL PROTECTED]
Department of Psychology http://www.psych.appstate.edu
Appalachian State University
Boone, NC 28608
USA
---------------------------------------------------------------
---
To make changes to your subscription contact:
Bill Southerly ([EMAIL PROTECTED])
