i went to some of the sites given in the urls ... and, quite frankly, it is
kind of difficult to really get a feel for what has transpired ... and how
targets were set ... and how goals were assessed
regardless of whether we like this kind of an approach for accountability
... or not ... we all have to admit that there are a host of problems with
it ... many of these are simply political and policy oriented (in fact,
these might be the largest of the problems ... when the legislature starts
enacting regulations without a real good understanding of the
methodological problems) ... some are measurement related ... and yes,
some are statistical in nature
we do have components of this total process
1. there are tests that are developed/used/administered/scored ... in 4th
and 8th and 10th grades ... these are NOT the same tests of course ... so,
one is never sure what it means to compare "results" from say the 8th grade
to the 4th grade ... etc.
2. then we have the problem that one year ... we have the data on the 4th
graders THAT year ... but, the next year we have data on the 4th graders
for THAT year ... these are not the same students ... so any direct
comparison of the scores ... 4th to 4th ... or 8th to 8th or 10th to 10th
... are not totally comparable ... so, ANY difference in the scores ... up
or down ... cannot be necessarily attributed to improvement or lack of
improvement ... the changes could be related and in fact, totally accounted
for because there are small changes in the abilities of the 4th graders one
year compared to another year ... (or many other reasons)
3. as i said before, we also have the problem of using aggregated
performance ... either averages of schools and/or averages for districts
... when we line them up and then assign these quality names of very high,
high, etc.
there is a necessary disconnect between the real goals of education ...
that is, helping individual kids learn .. and the way schools or districts
are being evaluated ... when averages are being used ...
4. i would like to know how on earth ... these 'standards' for
"dictated" improvement targets were derived ... did these have anything to
do with real data ... or an analysis of data ... or, were just roundtabled
and agreed to? we have to know this to be able to see if there is any
connection between policy targets and statistical problems
5. we have to try to separate relative standing data from actual test score
gain information ... and we don't know how or if the ones setting the
standards and making decisions ... know anything about this problem
so, to summarize ... there are many many issues and problems with
implementing any system whereby you are trying to evaluate the performance
of schools and districts ... and, perhaps the least important of these is a
statistical one ... set in the context of political policy matters ... that
a legislature works with ... and "legislates" targets and practices without
really understanding the full gamut of difficulties when doing so
unfortunately, in approaches like this, one makes an assumption that if a
school ... or district ... gets better (by whatever measure) ... that this
means that individual students get better too ... and we all know of course
that this is NOT NECESSARILY TRUE ... and in fact we know more than that
... we know that it is NOT true ... in many cases
sure, it is important to make some judgements about how schools and
districts are doing ... especially if each gets large sums of money from
taxpayers ... but, the real issue is how we work with students ... and how
each and every one of them do ... how each kid improves or not ... and, all
these approaches to evaluating schools and districts ... fail to keep that
in mind ... thus, in the final analysis, all of these systems are
fundamentally flawed ... (though they still may be useful)
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
