In article <[EMAIL PROTECTED]>,
[EMAIL PROTECTED] (Robert J. MacG. Dawson) wrote:
<Snip>>
>
> Wait one... Regression to the mean occurs because of the
_random_
> component in the first measurement. Being in an urban center is not
part
> of the random component - those schools' grades didn't improve because
> some of them woke up one day and found that their school had moved to
a
> wealthier district.
> If the effect of nonrandom components such as this is large
enough
> (as I can well believe) to justify the generalization highlighted
above,
> and if there was a strong pattern of poor-performing schools meeting
> their targets and better-performing schools not doing so, we are
looking at
> something else - what, I'll suggest later.
<Snip>
I do believe that regression to the mean is involved here. Each one of
the 1539 in the state is being evaluated. The performance by the 1998
group of 4th graders is being compared to the mean of the 1999 and 2000
4th graders. Every school is expected to improve, with the top
performing schools being expected to improve by an average of 2 points
on the scaled score. The scaled score is an odd duck, ranging from 200
to 280. There are a core group of questions in the 1998 to 2000 exams,
so that different students scoring the same number of correct responses
on these core questions would get the same individual score from year to
year. Schools are being "Failed" on the basis of 1 and 2 point
differences between years on this 200 to 280 point scale. In some
cases, it appears that a 2-pt difference might be due to just a
difference of 1 or two correct answers on the exam.
The school results are presented in a very odd fashion, making it
difficult to assess the patterns.
http://www.doe.mass.edu/ata/ratings00/SPRPDistribTables.html
The results for the 140 high performing schools are excerpted below,
with the target differences between the 1998 and @avg(1999:2000) exam.
Failed to meet Approached Met Exceeded Total
Diff Less then 0 0 to 1 1 to 3 > 3.1
35 18 36 51 140
The key to this table is found in the DOE rating guide.
http://www.doe.mass.edu/ata/ratings00/rateguide00.pdf
I'm not at all certain that this table isn't what I'd expect to see by
chance alone if there were an improvement of exactly the sort that the
Dept of Education was hoping for (a mean 2 point improvement). Rather
than reporting that the mean had improved, this Dept. of Education
report emphasizes failure. Schools in this high category were expected
to improve by 2 points on the 200 to 280 point MCAS scale. A high
performing school might have an overall score of 250 in 1998 on this
scale. If the mean of the 1999 and 2000 exams was less than the 1998
score, the school failed. If the difference was between 0 and 0.9, then
the school "Approached" its goal. If the score was between 1 and 3,
then the school "Met" its goal. If the difference in scores was greater
than 3, then the school exceeded its MCAS goal. I think it is more than
coincidence that the "Approached" category is half the "Met" category
since the bin size is twice as large for "Met" as "Approached".
For the lowest performing schools, their expectation was that they would
improve by 6 points. Any difference from 1998 and the mean of 1999 and
2000 less than 4 points on the 200 to 280 point scale earned an "F."
Schools approached their targets by improving between 4 to 5 points.
Schools met their targets by improving between 5 to 7 points and schools
exceeded their targets by improving by more than 7 points.
There were 91 critically low schools based on the 1998 exam, and here is
how they met their 6-point increase goal:
Evaluation Failed Approached Met Exceeded Total
Difference < 4 4 to 4.9 5 to 7 > 7.1
83 4 3 1 91
I frankly don't know what has happened with these MCAS scores. The
Dept. of Education's documents are sorely lacking in anything that looks
like valid statistical analyses, showing changes in means and standard
deviations. Instead of presenting valid, standard statistical analyses
to show what has happened to the mean scores, the state is reporting to
the general public an ordinal evaluation that makes the populace feel
that the students and their educators are failing them at all levels.
The top schools are getting smeared with failing grades when the results
may be consistent with random variation around a general pattern of
improvement, "the regression to the mean fallacy." Real improvements in
poor-performing schools may be masked by this attempt to convert scale
data into a poorly implemented ordinal scale of improvement.
--
Eugene D. Gallagher
ECOS, UMASS/Boston
Sent via Deja.com
http://www.deja.com/
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
