Tipsters, FYI Course Evaluations
Excerpted From:
The New Crisis in Teacher Evaluation: The Improper Use of 'Research-based'
Indicators. Professional Personnel Evaluation News. (January).Scriven,
Michael. (1988).
In many districts and states, the first crisis in teacher evaluation came
about not long ago when it was realized that the system being used - where
there was one - was wholly inadequate. To meet this crisis a gallant
response was generated, as witnessed by the oft-quoted figure of 40+
states installing new systems in the recent past or near future.
The second crisis arises from the fact that essentially all of these new
systems, and of those remaining unchanged, appear to be invalid. They are
invalid for essentially the same kind of reason that the evaluation of
personnel by the color of their skin or their church affiliation is
necessarily invalid. While it is true that much racial prejudice, sexism,
etc., is based on false beliefs about the groups discriminated against, the
essential flaw in it goes deeper than that The essential flaw is that even
if women in general are less strong than men, you shouldn't use gender to
discriminate against a particular candidate for a position as a
luggage-handler, but only a job-related strength test or series of
observations in a trial period on the job. And this is nor just for
ethical/legal reasons, but also for scientific reasons and reasons of
efficiency.
The epithet we sometimes use here for the political version of the wrong
approach is 'guilt by association'; the point is that we shouldn't make
decisions about an individual's merit on the basis of evidence about others
who share some characteristic with that individual. Which means you can't
discriminate against a teacher on the grounds that s/he exhibits some
approach to teaching that research has shown is less likely to be
successful. Whites are statistically less likely to be good basketball
players than blacks, but you can't kick the whites off the squad the day
you discover that the statistics are worse than you thought. nor would you
be any good as a coach is you used skin color as a criterion for selection.
You have to look at the individual's success, not at the success of groups
to which the individual belongs.
The current procedures for teacher evaluation involve exactly this fallacy
of using statistical-indicators and not just individual records, and they
are as certain to crash in the courts-eventually-as the most blatantly
racist hiring practices. We may have only a short breathing space before
the courts and defense attorneys begin to see the underlying similarity of
these two approaches. The consequences for states and districts will be
chaotic; old decisions may be reversed on appeal, huge damages may be
awarded, those hearings will clog the system, and there will be no
Legitimate process to take the place of the illicit one. (it is because of
this potentiality for disaster that we are giving a longer-than-usual
treatment of the issue here. More details will be found in an article by
the present author in the first issue of the Journal of Educational
Personnel Evaluation referenced earlier in this issue.) What has happened
recently is that in the rush to reform bad procedures for teacher
evaluation there has been an understandable move from the informal
procedures in wide use - which all too easily allowed personal biases to
enter - to procedures focusing on "validated" or "research-based"
indicators. These are indicators which research has supposedly shown are
positively correlated with successful learning by a teacher's students.
Popular envies are structured presentations, active involvement, emphasis
on positive reinforcement, high eye contact, high frequency of question
asking, provision of learning objectives, frequent feedback, use of
multi-media. The presence or absence of these factors defines a "teaching
style." and the argument here is that any reference to style is improper
in teacher evaluation,whether or not there is a research basis for thinking
the style is correlated with success.
Style indicators are not the only ones we can look at. We can also look at
the quality of the content of materials written, spoken, or distributed by
a teacher, at punctuality; at the relationship between grades awarded to
students and the quality of the work submitted by them; and so on. These
indicators relate to the professional performance of the duties of
teaching and they have a wholly different status from style indicators.
Performing those duties well is not just highly correlated with some global
definition of good teaching, it is good teaching. Here we call them
criteria of good teaching, by contrast with ("merely statistical" or
"secondary") indicators of it. There is no need for empirical research to
validate criteria; one needs only to look at job descriptions,
institutional mission statements, and obligations implicit in the role of a
teacher ("professional duties"). As in the case of racism and other
classical forms of discrimination, there is also a real problem about the
soundness of the claimed factual basis for the "research-based" approach,
but we are passing over that for the moment; perhaps we'll look at it in a
later issue. The main problem is that it doesn't matter whether the
research studies which claim to show certain indicators are correlated with
teaching success are any good. Regardless of their merit, the indicators
they identify can't be used in personnel evaluation. Just as in the case
of racism, that's not because, or not primarily because, of any ethical or
legal constraints, it's simply because it's Logically and scientifically
incorrect and therefore ethically and legally improper.
This is not to deny the validity of statistical inference. Useful
information is contained in a statistical correlation, and there are
circumstances in which that information can
be put to good use. It can even be put to good use in making decisions
about people - but only when no better data is available because of
limitations on time or resources. As the Titanic sinks we may with some
justification fail back on the selection principle of "women and children
first" which uses secondary indicators of merit, because they can't be
instantly applied, make the appropriate-sized cut, and are backed by
well-known and non-trivial lines of argument. If the basketball (or
football or track) coach is forced to select a team given only the race of
the candidates, s/he will certainly do better - although not very well -
to pick blacks. In real world personnel evaluation, however, such
emergencies are avoidable and the low-quality decisions that they support
can be improved considerably by looking at individual "track record" data.
Moreover - a crucial point that we'll justify later - once we have better
data, it renders the low-specificity data completely irrelevant; we can't
add it in to get further improvement in the selection process.
For these reasons, if the relevant decision-maker - a personnel officer,
whether coach or principal or dean - does argue that no other data is
available and that we must perforce (or ideally) use the kind gathered by
most of the instruments now used (or that we should additionally use that
data), we should normally conclude that claim constitutes a proof of
incompetence so conclusive as to provide grounds for dismissal of the
personnel officer. Thus the one personnel decision that can be based on the
use of statistical indicators is the dismissal of the person who uses
them.......
_______________________________
_________________________________
_________________________________
+<[#:->[>>
John C Damron, PhD *
Douglas College, DLC * * *
P.O. Box 2503 * * *
New Westminster, British Columbia
Canada V3L 5B2 FAX:(604) 527-5969
e-mail: [EMAIL PROTECTED]
FAX (604) 527-5960 Tel: (604) 527-5860
<<<<<<< ================== >>>>>>>
http://www.douglas.bc.ca/
http://www.douglas.bc.ca/psychd/index.html
