On Thu, 29 May 2008 12:50:40 -0700, Ken Steele writes: >A review article of these issues has appeared in timely fashion- > >Sackett, P. R., Borneman, M. J., & Connelly, B. J. (2008). >High-stakes testing in higher education and employment: Appraising >the evidence for validity and fairness. American Psychologist, 63 (4), >215-227.
I thank Ken for bringing this article to our attention and I have only started to skim it but I'm already getting headaches. I want to give it a fair reading but I already see that I'm going to have to dig up cited references. Unfortunately, some of them, such as the Berry and Sackett (2008) study cited on page 218 is a conference presentation and I'll probably have to write to Sackett to get a copy of the prsentation (I would also like to get a copy of his dataset and run my own analyses on it). But I'd like to raise a larger issue and get feedback on the it. One of the key problems in analyzing the relationship between the SAT and measures of academic performance, such as GPA at the end of the first year, is that of range restriction: colleges usually do not admit people with SAT scores belows a certain value so the range of values for both the SAT and GPA are both restricted. However, consider the following two situations: (1) Imagine we measure two characteristics of a population of individual, say height and weight. We expect there to be some nonzero population correlation between these two variable across that entire range of possible values for the two variables. But consider we have some selection process that only allows people above a certain height to be included in a sample. We now calculate the correlation for this sample and not surprisingly, the correlation between height and weight in this sample is smaller in the sample relative to the population. The reduction in the value of the correlation is due to restriction of range and there should be little controvery about the comparison. (2) Imagine that we have two variables, assume that both might be normally distributed and that we can calculate a correlation between the two variables. The problem is variable X (say SAT scores) has a peculiar relationship to variable Y (say GPA scores) namely Prob (Y | X greater than a critical value) > 0 and Prob (Y }X less than a critical value) = 0. For example, a college only admits students with a combined SAT Quantitative and SAT Verbal score of 1200. So, it is only possible for have GPA grades for combined SAT >= 1200 but impossible to have a GPAs for combined SAT < 1200. Theoretically, the combined SAT distribution is a truncated normal distribution which has mathematical properties different from ordinary normal distributions. The nature of this situation is such that, becuse it is impossible to get GPAs for combined SATs below 1200, the proper correlation is between combined SAT >= 1200 and GPA. Any correction for range restrictions refers to a "fantasy" situation which cannott exist. Correlations based on corrections for range restriction, therefore, are fantasy correlations because they require assumptions that are false in reality (i.e., if the college accepted students with SATs below 1200, what would their GPAs be? But these people and values don't exist -- indeed, would the college remain unchanged if it admitted students with combined SATs < 1200?). So, in situations like that described in (1), it seems reasonable to make range restriction corrections to correlations because in reality the entire range of values exist for both variables. In situations like that described in (2), it seems *unreasonable* to make range restriction corrections to correlations because it requires one to accept the counterfactual assertion "If we had accepted students with combined SATs below 1200 we expect the correlation in the trucated situation to extend to this larger sample". But the larger situation doesn't exist. And if the college did allow in students with combined SATs below 1200, it may no longer be the college it was prior to the change (e.g., change in school philosophy along with increases in support services for students, etc.). Consequently, using range corrected correlations in such situations are essentially meaningless. If this is accepted, then a number of correlations that Sackett et al (2008) report may be of dubious value. Comments? I'd just mention that this thought isn't original with me though I don't know whether others have applied it to the issue of range-corrections to correlation. I don't have a copy of Snedecor & Cochran's Statistical Methods handy but I do recall an argument that was made in the context of the analysis of covariance. Using covariates to control for various variables in an ANOVA of an experiment might be problematic if the removal of such effects is impossible in real life. The ANCOVA may produce statistically significantly results but the results may not generalize to the real world. If any one has a copy of S&C, could they check on this, just in case I'm dealing with a false memory or temporary psychotic break with reality. ;-) -Mike Palij New York University [EMAIL PROTECTED] P.S. Anyone know any studies showing the correlation between SAT and either time to completion of degree requirements or dropout? --- To make changes to your subscription contact: Bill Southerly ([EMAIL PROTECTED])
