This post is to clean up a few dangling threads and to correct an error
in my previous post.
In response to Rich, I pointed out yet another potential problem in the
MCAS. School effectiveness is being based on the mean of scaled MCAS
scores, which range from 200 to 280. Rich pointed out that the odd
scaling might have been intentional, to limit erroneous ratio-scale
comparisons, such as "My scores are twice as good as yours!" In
thinking about this problem, I recalled Fred Roberts' discussion of
interval and ratio scales in his book "Measurement Theory" and in his
book "Discrete Mathematical Models." Roberts pointed out that degrees
Kelvin is a ratio-scaled variable, but degrees F and degrees C are only
interval-scaled. The MA Dept. of Education's method for converting raw
to scaled scores produces a scaled score that preserves the ordinal
ranking, but the transformation does not appear to be a valid
transformation to preserve the interval scaling. If a variable isn't
measured on an interval scale, than calculation and comparison of means
is meaningless in Roberts' terminology.
In the following technical report for the 1999 exam, the Dept. of
Education provides the equations to convert raw scores for one exam in
one year, usually based on 1 point for getting one correct answer, to
the 200 to 280 scale:
http://www.doe.mass.edu/mcas/2000docs/pdf/99techrep.pdf
On page 70 of this 112 page pdf (p 66 using the report's pagination),
two regression equations are provided for converting raw scores, r, to
scaled scores, S, for this exam:
Scaled score = 1.59 raw score + 177.25 if raw score <38.83
Scaled score = 2.65 raw score + 136.19 if raw score >38.83
Now, the 38.83 is the raw score that a panel had decided was the cutoff
for proficiency in this subject area in this year. So, if a student is
proficient, every point scored adds 2.65 points to the scaled score. If
the student misses the proficient cutoff of 38.83, a point on the exam
is worth only 1.59 scaled points.
This scaling approach seems analogous to measuring temperature with two
different sets of thermometers, which happen to both record the same
temperature at only one point. I miscalculated that point for the F and
C scales in my previous post. It is -40 degrees. So here is my
hypothetical MCAS question rephrased:
To measure the temperature in a freezer, a scientist has two
thermometers, one Fahrenheit and the other Celsius. When the temperature
is less than -40 degrees, the scientist uses the Celsius scale. When
the temperature is greater than -40 degrees, the scientist uses the
Fahrenheit scale. In 1998, the scientist took 40 measurements of the
temperature in the freezer. In each of the following two years, he took
40 more measurements of the temperature in the freezer. Overall, the
number of measurements using the Celsius scale was between 25% and 50%
of the total number of measurements.
The mean temperature was one degree less in 1998 than the mean of the
1999 and 2000 temperatures.
Should the scientist:
a) Conclude that the temperature in the freezer is getting colder
b) Conclude that the temperature in the freezer is about the same
c) Conclude that the temperature in the freezer is getting warmer
d) Not conclude anything since it is meaningless to average degrees
Fahrenheit and degrees Celsius in this way.
--
Eugene D. Gallagher
ECOS, UMASS/Boston
Sent via Deja.com
http://www.deja.com/
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
