Posted by <a href="http://www1.law.ucla.edu/~sander/";>Rick Sander (guest-blogging)</a>: Responding to Critics (3): Selection-Bias Blues http://volokh.com/archives/archive_2005_06_12-2005_06_18.shtml#1118799904
Iāve posted in this space data that shows blacks who pass up the best law school that admits them, and go to their āsecond-choiceā school, are closer in credentials to their classmates and have much better outcomes during and after law school. The postings have generated much discussion. Professor Dirk Jenter, while defending me from the social science nihilism of āMahan Atmaā, offers a pointed critique of the āsecond-choiceā analysis: isnāt the analysis contaminated by self-selection? The students going to their second-choice schools are, of course, doing so consciously; maybe that means theyāre a group that believes they will optimize performance at a less elite school, which makes their subsequent, superior performance at those schools and on the bar exam less surprising. Selection-bias problems are an ever-present danger in this type of observational data, creating pitfalls which more than one of my critics have fallen into. It is probably not possible to eliminate entirely all danger of selection-bias in this comparison of first- and second-choice students, but I am pretty confident that thereās little or no such bias here, for several different reasons. (See my [1]"Reply to Critics" for a fuller discussion) First, these students chose responses indicating that financial or geographic factors led them to turn down their first choice school and go somewhere else. And their other answers to the detailed surveys they completed were consistent with those answers ā although they cared about school āelitenessā almost as much as other students, they cared about ācostā and āfinancial aidā a lot, too. So, the motivations of these students didnāt seem related to some kind of strategy of seeking out a less competitive environment. Second, we have a wealth of data about the strategies of these students as they started law school; in every way Iāve been able to measure, they seem to be approaching law school with strategies and expectations that are indistinguishable from all the other black students. For example, both the second-choice and other students are equally likely to respond that they are āvery concernedā about getting good grades in law school (89% vs. 88%), and both groups are equally likely to think they are going to end up in the top tenth of their law school classes (37% vs. 38%). Blacks in general express more concern in the survey data about passing the bar ā but, ironically enough, both black going to second-choice schools and all the other blacks tend to think that going to a more elite school will improve their chances on the bar. All of this data points against selection bias. Third, it is important to keep in mind that this entire exploration of the āsecond-choiceā phenomenon is a way of confirming the hypotheses I developed and tested with entirely different data in my original article. I didnāt observe this high performance among blacks going to second-choice schools, and then construct a theory around it; this data was brought to light by others after Systemic Analysis had gone to press. In Systemic Analysis, Iām comparing blacks (as a group that generally is boosted into more elite schools by racial preferences) against whites (who sometimes receive preferences, but generally donāt), while controlling for entering credentials. Certainly thereās no self-selection process there (or only a little, accounting for students with mixed-race backgrounds). Whatās nice about the first-choice/second-choice analysis is that it avoids arguable pitfalls of the white/black analysis, and vice versa. But both methods produce essentially identical results. āMichaelā raises another interesting issue. In estimating the average ācredentials gapā facing blacks at their second-choice schools (and comparing that with the credentials gap facing other blacks), I use the six loosely-defined ātiersā in the LSAC-BPS database. The creators of this database grouped schools into āclustersā by using some indicators of prestige (e.g., student scores) and some indicators unrelated to prestige (e.g., public sector vs. private sector). The six tiers certainly correlate substantially with school prestige, but they also undoubtedly overlap. So the most elite Tier 2 schools are almost certainly higher-ranked than the least elite Tier 1 schools, even though Tier 1 as a whole is clearly more elite than Tier 2 as a whole. Consequently, one needs to be careful about using the tiers in sensible ways. In the second-choice analysis, I know each studentās grades (standardized by school) and individual outcomes (e.g., graduation), but in terms of school identity I only know what tier they are in. I compared each studentās index score to the median index score of students in the same tier to estimate the typical credentials gap between students and their classmates. This is, of course, a rough measure ā but the key concern is whether thereās some reason to think itās biased in a way that helps my analysis. The answer is: I donāt think so. Blacks other than the second-choice students should be distributed more or less randomly across the six tiers. The blacks going to second-choice schools should also be pretty randomly distributed ā with one exception. In Tier 1, they are more likely to be in schools near the bottom of that elite tier rather than at the top (since they have generally rejected a more elite school, they are unlikely to be going to Harvard or Yale). But that distortion would mean I was overstating their actual mismatch with their fellow students, which cuts against my analysis, not in favor of it. And, in any case, for the whole group of students this is likely to be a small distortion indeed. Consider, by way of contrast, an analysis done by Dan Ho (a recent Yale graduate and Harvard Ph.D. whose critique, and my reply, are in the June issue of the Yale Law Journal). Ho compared blacks who had the same index scores but attended schools in adjacent tiers. Hoās argument went like this: if two blacks with the same entering credentials went to adjacent tiers, and passed the bar at the same rate, then thereās no penalty to blacks from going to higher tier schools, and Sander is wrong. But of course, if the BPS tiers overlap, then for any analysis which selects blacks from, say, Tier 1 and Tier 2 schools who have matching index scores, itās quite likely that these students are actually going to schools of equivalent eliteness. Here the bias is likely to be quite significant, and it all is in the direction of the results Ho wants. Hoās analysis is invalid for other demonstrable reasons, but I offer it here as an example of an improper use of tiers. A final point: āMichaelā also asks whether we know the blacks going to their second-choice schools actually got into their first-choice school. The answer is yes, we do ā the questionnaire asks that question, and we used it as a filter. Tomorrow Iāll address some critiques of [2]Systemic Analysis itself. References 1. http://www1.law.ucla.edu/~sander/ 2. http://www1.law.ucla.edu/~sander/ _______________________________________________ Volokh mailing list Volokh@lists.powerblogs.com http://highsorcery.com/cgi-bin/mailman/listinfo/volokh
