At 11:31 PM 1/10/01 -0500, Bob Hayden wrote:
regression to the mean applies to relative position ... NOT raw scores
let's say we give a test called a final exam at the beginning of a course
... and assume for a moment that there is some spread ... though the mean
necessarily would be rather low ... then, we give an alternate form of this
final exam at the end of the course ... where again, there is reasonable
spread but, obviously, the mean has gone up alot ...
now, EVERYONE'S SCORES GO UP ... so everyone improves ... and it is not
that the low scores (BECAUSE of regression) will improve more and the
better scoring students on the pretest will improve less) that is NOT what
regression to the mean is all about ...
so, it depends on how these tests are scored and reported ... if the scores
are reported on something like a percentile score basis ... then there is
necessarily a problem ... but, if the scores are reported on some scale
that reflect that 10th grade scores are higher than 8th grade scores ...
and 8th grade scores are necessarily higher than 4th grade scores ... that
is, the scores reflect an ever increasing general level of knowledge ...
then regression to the mean is not the bugaboo that the "letter" makes it
out to be
now, the post said:
The effectiveness of school districts is being assessed using average
student MCAS scores. Based on the 1998 MCAS scores, districts were
placed in one of 6 categories: very high, high, moderate, low, very low,
or critically low. Schools were given improvement targets based on the
1998 scores, with schools in the highest two categories were expected to
increase their average MCAS scores by 1 to 2 points, while schools in
the lowest two categories were expected to improve their scores by 4-7
points
=========
there are a number of ? that this paragraph brings to mind:
1. how are categories of very high, etc. ... translated into 1 to 2 points
... or 4 to 7 points? i don't see any particular connection of one to the other
2. we have a problem here of course that the scores in a district are
averages ... not scores for individual kids in 4th, 8th, and 10th grades
3. what does passing mean in this context?
4. let's say there are 50 districts ... and, for last year ... using 4th
grade as an example ... we line up from highest mean for a district down to
lowest mean for a district .... then, in the adjacent column, we put what
those same districts got as means on the tests for the 4th grade this year
....
we would expect this correlation to be very high ... for two reasons ...
first, means are being used and second, from year to year ... school
district's population does not change much ... so if one district has on
average, a lower scoring group of 4th grade students .... that is what is
going to be the case next year
thus, given this ... we would NOT expect there to be much regression to the
mean ... since the r between these two variables i bet is very high
5. but, whatever the case is in #4 ... what does this have to do with
CHANGE IN MEAN SCORES? or changes in the top group of at least 1/2 points
and in the low groups changes of 4/7 points? the lack of r between the two
years of 4th grade means on these test just means that their relative
positions change with the higher ones not looking as relatively high ...
and the low ones not looking quite so relatively low BUT, your position
could change up or down relatively speaking regardless of whether your mean
test performance went up or down ... or stayed the same
bottom line: we need alot more information about exactly what was done ...
and how improvement goals were defined in the first place ... before we can
make any reasonable inference that regression to the mean would have
anything to do with better districts being bad mouthed and poorer
performing districts being praised
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
