I have an encountered an unusual but important problem in calculating the 95% confidence intervals for the 10th grade high stakes MCAS exams in Massachusetts. I plot the 95% confidence intervals on pages 8 and 10 on this 12-p pdf, which describes the problem. http://www.es.umb.edu/edg/MCAS/mcaserrors.pdf
Thousands of MA seniors are being denied diplomas this week because their scaled MCAS scores did not meet the 220 cutpoint on both the Math and English tests. This failure cutpoint in 2001 was 20 correct answers (out of 59) on math and 39 correct answers (out of 71) on English. Many of these students have scores of 216 and 218 on the 200-280 MCAS scale. The MA Dept of Ed reports the error on this exam as about +/- 1.5 points for students in the 216-218 range. This is +/- 1 Standard Error. However scores just above the 220 failure cutpoint have standard errors of 3-6, producing very large 95% confidence limits, which overlap the failure cutpoint for scores up to about 230. The reason for this is that a correct answer which produces 1 additional point on the raw scale below the 220 cutpoint adds only 0.5 points to the ELA MCAS score (on the 200-280 scale). An additional correct answer on the English exam raw scale is converted to 2 additional points above the 220 failure cutpoint. For the math exam, raw points count about 1.0 and 1.6 MCAS scaled points above and below the 220 failure cutpoint. The change in the linear transformation used to scale scores produces the link-sausage-shaped confidence limits, with very narrow confidence limits in the 210-220 range (see Fig. 2 & 3) I consider these narrowed confidence limits to be an artifact of the scaling. If a straight linear scaling is used for the conversion, the confidence limits would remain broad for those scores in the 210-220 range. DOE uses a complicated scaling procedure to generate their standard errors (described in their just-released 2001 technical document). The standard errors are based on calculations done on the raw MCAS scale. DOE reports the standard error for math scores as 3.68 points on the raw math scale (0 to 59 points) and 3.59 points on the raw English scale (0 to 71 points). It is these standard errors that are translated to values that are about 1.5 standard errors for MCAS scores just below the 220 cutpoint and standard errors of 3-6 just above the 220 failure cutpoint. I know that reading a 12-p pdf is much to ask. Just looking at Figs 2 and 3 and providing comments would be appreciated. The Boston Globe has just completed a 4-article series on MCAS. I have forwarded my analysis to one of the Globe reporters in the hopes that they will cover this before students are denied the right to even walk across the stage at high school graduations this weekend. Gene Gallagher, UMASS/Boston . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
