Hi Mike's pessimistic view of grades and tests does not jibe with my experience. Just a couple of observations.
1. Whenever I have examined Cronbach's alpha for my multiple-choice or short-answer tests I invariably find quite respectable values. So such tests are certainly not a lottery in the sense that students' answers to each question are like a coin toss. 2. My independently scored multiple-choice and short-answer marks invariably correlate with one another, albeit far from perfectly, of course. But unreliable measures should not (can not?) show such consistency with one another. 3. If individual assessments were simply noise, then all students would end up with the same final average, especially in courses with numerous assessments. And across all courses, their gpas would be about the same. This is clearly not the case. Indeed a problem in many courses is exactly the opposite ... a bimodal distribution. And a corresponding problem at the aggregate level is students who cannot maintain an adequate gpa. 4. My admittedly subjective judgement of the students I get to know best (i.e., honours students) is that the excellent students are clearly superior to students with lower grades, even when the difference is marginal (e.g., A+ vs. A vs. A-). The papers, presentations, whatever of the top students are just superior. And the fact that they stand out in class after class again indicates the consistency of this judgment across courses and faculty. I would be very surprised if a blind marking of essays by students of different grade levels did not provide validation of the grades. 5. Restriction of range is clearly a problem in evaluating predictors at the university level, especially at selective institutions. A colleague was talking at lunch today about the French system. University is free and very many students attend. He also observed that very many tend to drop out in the first few years (he mentioned 80% ... perhaps this is the model described by Chris in effect). I would bet a fair amount that a French equivalent of the SAT would be highly predictive of who would drop out. 6. Even accepting the modest existing correlations, however, caution is needed. While it is true that a small correlation can be significant given a large enough n, it is not true that a small correlation (effect size) is necessarily unimportant. The classic example is the aspirin study ... a minuscule effect translated into many lives saved because of the huge numbers involved. Similarly huge numbers are involved when it comes to universities as well, and (like aspirin) the cost of the test is low relative to the cost of a year of university, both for the institution and for the student. I would be very interested in evidence that grades or objective tests of aptitude/ability/achievement are "like a lottery." Take care Jim James M. Clark Professor of Psychology 204-786-9757 204-774-4134 Fax [EMAIL PROTECTED] >>> <[EMAIL PROTECTED]> 29-May-08 12:42:29 AM >>> By the way, in both groups, the first year GPA was better predicted by HS rank than SAT but SAT score contributed a significant amount of prediction over and above the class ranking when combined in multiple regression. Bill Scott What were the effect sizes? How much variance was explained by the model? This is the issue neglected in all these reports. I can get a statistically significant multiple R of .4 with the sample sizes in these studies. Would you base a financial decision on a model with that level of error? The major problem with accepting everyone and then paring down the classes each year based on performance is that it ignores competence and it ignores the fact that grades have unknown validity and reliability. It may be the case that all the students are actually within the same range of competence and the differences between them are accounted for by error in using grades to stratify them. This commonly happens in medical schools. We accept these super-students who all do extremely well in all their courses. Since we have to enforce a Bell Curve when none exists, their tests are manipulated by their teachers who generate a set of items that actually incorporate random error that looks like valid performance variance. A very tiny number of medical students do poorly in classes. They get jerked around by their instructors because the instructor looks bad if everyone gets an A or B. It is completely logical for everyone in a class to get an A (or an F). In particular, 98% of medical students actually perform in the range of an A. We have hyper-selected them for a bizarre level of study and test-taking skill. Unfortunately, the faculty cannot tolerate this. They design their test responses so ambiguously that a student who truly knows the correct answer cannot find it among the options. If we were to empirically terminate students based on some percentage each year, we would be terminating a number of successful students who are only being selected for failure because of the error in our tests and grading systems. The issue of inflation of High School grades is largely irrelevant. They don't predict college performance very well. We could only use them as a very general screening tool. We use them and SAT scores as if these things measure something important in a reliable and valid manner. We know in our hearts that they do not. As a consequence, we tell lies to students who are accepted and to those who are rejected. What we should tell them is that the selection was largely random either way. I think we should be completely honest and have fair lotteries if we need to restrict admission. Otherwise, financial concerns, legacies, idiosyncratic foolish ideas and all the weak aspects of human reasoning, such as confirmation bias, determine the selection. The fact that that most students, parents, admissions offices and faculty are not aware of the random error involved in the process is a reflection of this. We present a facade of validity and fairness when the process is actually at best, a lottery. Mike Williams http://www.learnpsychology.com --- To make changes to your subscription contact: Bill Southerly ([EMAIL PROTECTED]) --- To make changes to your subscription contact: Bill Southerly ([EMAIL PROTECTED])
