Dear Gene,
I share your feelings about MCAS (fortunately my daughter finished high
school before it came into effect, but that's no consolation for the
thousands of other kids who are supposed to be deprived of their high
school diplomas), and had some similar reactions to the article.
On the statistical issue, there are two ways that this could be done that
have different implications. If you form categories based on the actual
score in the previous year, then you have a classic regression to the
mean situation. The magnitude of the regression to the mean effect could
be estimated knowing variances and sample sizes at the various schools;
it is even possible that allowing for a few points smaller required
advance in the higher-scoring school might adjust for that effect
(although I would be surprised!).
Another approach would be to define groups by some other variable related
to but distinct from outcomes, e.g. inner city versus suburbs, percent
minority, or percent in poverty. Although the groups will differ in their
baseline values, there is no regression to the mean effect there.
However, either of these analyses begs the question of the potential for
improvement in different schools. An excellent school may already be doing
everything it should be doing and have no way to improve, while a low-scoring
school may have a lot of possible avenues to improvement. There may also
be issues about the appropriateness of the educational strategies that might
be adopted. Somebody might argue that at a school already doing well at
MCAS, improvements could only be obtained by teaching to the MCAS at the
cost of less investment in AP exams. (I personally have little confidence
in the educational incentives of the MCAS at any level; I am just laying
out possible arguments.) Presumably some arguments could also be advanced
to the opposite effect, as well.
Best regards
Alan Zaslavsky
P.S. I see now that Robert Dawson had already made some similar comments,
which I hadn't gotten to yet when I wrote this.
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
