Alan Zaslavsky wrote:
>
> The following article may be of interest to some of you who are trying to
> get across the notion of reliability, particularly those who are teaching
> H.S. or young college students who have recently gone through high-stakes
> achievement/competency testing programs. You can also download directly
> from the New York Times web site at
>
> http://www.nytimes.com/2000/09/13/national/13LESS.html
...
> Not really. David Rogosa, a professor of educational statistics at
> Stanford University, has calculated the "accuracy" of tests used in
> California to abolish social promotion. (New York uses similar tests.)
>
> Consider, Dr. Rogosa says, a fourth-grade student whose "true" reading
> score is exactly at grade level (the 50th percentile). The chances are
> better than even (58 percent) that this student will score either above
> the 55th percentile or below the 45th on any one test.
>
> Results for students at other levels of true performance are also
> surprisingly inconsistent. So if students are held back, required to
> attend summer school or denied diplomas largely because of a single
> test, many will be punished unfairly.
>
> About half of fourth-grade students held back for scores below the 30th
> percentile on a typical reading test will actually have "true" scores
> above that point. On any particular test, nearly 7 percent of students
> with true scores at the 40th percentile will likely fail, scoring below
> the 30th percentile.
>
> Are Americans prepared to require large numbers of students to repeat a
> grade when they deserve promotion?
...
The obvious question is: what are the actual standards for promotion?
What does "deserving" it mean? [The author of the piece appears not to
question the value of promotion - even beyond one's level of competence
- or consider the possibility that it might be harmful to those who are
in fact not ready. This of course only makes sense if one assumes that
schools are there to provide diplomas rather than education.]
The article _suggests_ that the tests have no objective pass mark but
are set up so that, no matter how well students do overall, 30% will be
held back. Is this true (it should not be in a well-run system!), or are
the percentiles referred to just the percentiles currently corresponding
to pass marks that have some objective justification?
If the authorities indeed have no idea of what constitutes acceptable
performance, but have decided to hold back an arbitrary 30% of the kids
for the rest of time, no matter how good or bad performance at that
percentile might be, this is wrong and would not be made right even if
everybody's "true" score could be determined to eight decimal places.
If the pass mark is based on "acceptable performance" that has prepared
the student for the next grade, balancing costs and benefits would
presumably suggest that the pass mark be put somewhat below the level at
which one could say "this student is absolutely definitely ready to
proceed.", as setting it at or above that level would mean that no risk
whatsoever of passing an unprepared student was being taken. Therefore,
one could not say that a student whose "true" score was slightly above
the cutoff "deserved" promotion.
Whether the position of the pass mark relative to an "ideal pass mark"
is appropriate given the magnitude of the random component of a
students' score can only be determined by looking at actual performance.
Putting everything in terms of percentiles makes it impossible to even
ask such a question - and suggests that Dr. Rogosa may not have asked
it.
-Robert Dawson
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
