Hi Tips folks!.. Happy New Year!
Below is an assortment of comments contributed by members of faculty to
the well known (and somewhat celebrated!) Chronicle of Higher Education:
Colloquy--1998 -- cleaned-up and internet friendly. I suspect that they will
provide members of faculty with a rather useful resource when dealing with
and/or debating some student ratings issues.
- All the very best!
-----------
The January 16 Chronicle article on "Student
Evaluations" quoted one researcher as
saying, "With student ratings, you've got
someone who's sat through 40 hours of a
course." Similarly, among these Colloquy
responses, another writer opined that "It is
ironic that so many individuals believe that
students are not capable of providing
meaningful evaluative feedback or make
accurate reflections on their experiences."
These statements seem to imply that students
give accurate ratings. Relevant to this
proposition is the following empirical
demonstration provided by L. E. Stanfel
(1995, Journal of Instructional Psychology,
22, pp. 117-125):
"For Questions 1 ["The instructor's]
objectives for the course have been made
clear."] and 4 ["The instructor explains
clearly to students how they are
evaluated."] the vehicle of [documentation]
was an open-book, multiple-choice quiz ...
every student answered both quiz parts
correctly and thereby proved that they knew
both the course objectives and the procedure
by which they would be evaluated. ... For
Question 5 ["Tests and written assignments
are graded and returned in a reasonable
period of time."] a dated sign-up sheet was
circulated at each occasion when graded
documents were returned ... The original
documents' being dated, the student
signatures proved the graded documents had
been returned to class at the earliest
possible opportunity" (Stanfel, 1995).
For the first course reported by Stanfel
(1995), Question 1 received: 1 strongly
agree; 6 agree; 7 uncertain; 19 disagree;
and 6 strongly disagree. The data reported
allow the following calculations: on a
5-point scale, if 5 were the best score, the
average response to Question 1 should have
been 5.00; instead, the average response to
Question 1 was 2.41; the averages for
Questions 4 and 5, respectively, were 2.64
and 3.79; for the second course, the
averages for Questions 1, 4, and 5 were
1.88, 2.00, and 3.75, respectively. As
Stanfel notes, "since [responses to three
questions] have been proven patently
incorrect, it is impossible to suppose the
remainder [of responses to the
questionnaire] is any less erroneous"
(Stanfel, 1995).
This negative halo effect demonstrated by
Stanfel probably reflects the discriminant
invalidity which Greenwald and Gillmore
document (1997, American Psychologist, 52,
p. 1214). Potentially making matters worse,
Marsh and Roche (1997, American
Psychologist, 52, p. 1188), two enthusiastic
defenders of student evaluations of
instruction, speculate that "Global or
overall ratings ... may be more susceptible
to context, mood, and other potential biases
than are specific items that are more
closely tied to actual teaching
behaviors..." If that is the case, then
student responses to any global items on
Stanfel's (1995) questionnaire would have
been even more untruthful than those to the
specific items described above. So there you
have it: students are in a position to
provide accurate reports about their
classroom experiences, but the fact is that
in many cases they do not provide accurate
reports.
I would also like to comment on what one of
the writers in this colloquy refers to as
"forty years of very consistent research
demonstrating the underlying validity and
usefulness of student-provided data." Not
even counting the recent work by Greenwald
and others, consider the negative evidence
provided by several actual classroom
experiments such as that of Chacko (1983,
Educational Research Quarterly, 8 (2),
19-25):
"... Pre- and posttreatment measures of Ss'
perceptions of instruction were obtained
from an experimental and a control group of
Ss using a Likert-type scale consisting of
items describing commonly shown teaching
traits and behaviors. The 2 groups did not
vary in their premeasure scores. A midterm
examination was administered, then evaluated
in such a way that Ss in the experimental
group were more harshly graded than Ss in
the control group. Results show that the
ratings of Ss in the experimental group
changed to reflect more negative perceptions
of instruction. These shifts were
statistically significant in 7 of the 10
rating items. Such changes were not seen
in the control group" (Chacko, 1983).
Findings such as these, bearing negatively
on the validity of student ratings, have
been dismissed on spurious grounds by
unabashed advocates of student evaluations.
These advocates would gain a lot more
credibility if they would acknowledge that
student ratings have both validity and
invalidity and if they would constructively
address the invalidity problem. In the
meantime, the rest of the academic community
must be vigilant and resolved in the face of
Pollyannaish claims about student ratings
that can then be used to justify misuse of
ratings by administrators who are so
inclined.
-- James M. Puckett, Associate
Professor of Psychology, Texas A&M
University, Kingsville (posted 1/26, 12
--------------------------------------------
I have just two simple points with respect
to student ratings:
(1) Students are neither objective nor fully
informed. They can have axes to grind. They
do not know what is good teaching in the
discipline or whether the content is being
handled thoroughly or appropriately. That is
why they are not teaching the class.
(2) Higher education is the only part of the
"consumer economy" where the "customer"
expects less for his or her money than they
are paying. If education is merely
credentialization, then the student may
reasonably expect that any professor who
impedes or disturbs progress towards the
credential is doing him or her a disservice.
However, if learning is truly the student's
highest priority (instead of the 35 hours a
week job for minimum wage to pay for the car
and apartment and cable TV), the student
needs to assume primary responsibility for
learning (or not doing so). All the
professor can do is provide a suitable
structure to assist the student. That
structure must include sufficient rigor and
demands on the student's time and effort,
even if discomfort is produced.
--Anthony Duben, Assoc. Dean, Col. of
Science and Tech., Southeast Missouri
State Univ. (posted 1/30, 12:20 p.m.,
E.S.T.)
_______________________________________
It is hardly surprising that individuals
representing the evaluation industry within
academe (both those who produce instruments
and correlate the results, and those
administrators who use the products to
determine promotion and/or tenure) would
defend the validity, integrity, and goals
of their work.
Part of the problem is cultural, and part is
a combination of epistemology and ethics.
The cultural problem is that no matter how
much we may seek to produce value-neutral
means for measuring performance, inevitably
evaluation rankings confer actual or
perceived social status along with "neutral"
measurement. An "A" student has higher
status than a "C" student, and the same is
true with professorial evaluations. This
means that no matter how much individuals or
systems may seek objectivity, cultural value
systems inhibit these aims.
The ethical and epistemological issue
concerns the debate between process and
outcome as aims of education. In a tradition
extending from Aristotle to Dewey, it has
been held that the ultimate test of a
successful practice can come only at the end
of one's life. It is only in retrospect,
surveying the narrative of a whole life,
that one can truly know whether one has
lived a good life. Meanwhile, one immerses
oneself in the process of seeking the mean.
The measure of such is the "person of
practical wisdom"; a person and her
character, not some feeling state or
measurable performance level in knowledge or
skills.
Those who advocate outcome based assessment
of student learning as a major criterion for
teacher evaluation stand within the
utilitarian tradition of the greatest good
for the greatest number. They tend to
believe that quantitative evaluation of
outcome, in terms of goods such as pleasure,
is both possible and worthwhile; indeed,
they often believe that the only "real"
results are those that can be measured by
whatever criteria they have devised. They
are concerned with product rather than
process, with how to measure the
satisfaction of consumer desire rather than
the character of learners.
Utilitarianism is the default position of
middle management in industry for reasons
that are self-evident. It is not clear to me
that such ought to be the case with
professions in general and with teaching in
particular. But then, I believe that there
are rational, other-regarding, valid reasons
to defend the importance of developing
inter-personal character in cultural,
social, and political contexts, and some of
my colleagues in the social sciences do not.
-- David H. Fisher, Ph.D., Chair &
Professor of Philosophy (posted 1/16,
10:45 a.m., E.S.T.)
----------------------------------------
Those who advocate outcome based assessment
of student learning as a major criterion for
teacher evaluation stand within the
utilitarian tradition of the greatest good
for the greatest number. They tend to
believe that quantitative evaluation of
outcome, in terms of goods such as pleasure,
is both possible and worthwhile; indeed,
they often believe that the only "real"
results are those that can be measured by
whatever criteria they have devised. They
are concerned with product rather than
process, with how to measure the
satisfaction of consumer desire rather than
the character of learners.
Utilitarianism is the default position of
middle management in industry for reasons
that are self-evident. It is not clear to me
that such ought to be the case with
professions in general and with teaching in
particular. But then, I believe that there
are rational, other-regarding, valid reasons
to defend the importance of developing
inter-personal character in cultural,
social, and political contexts, and some of
my colleagues in the social sciences do not.
-- David H. Fisher, Ph.D., Chair &
Professor of Philosophy (posted 1/16,
10:45 a.m., E.S.T.)
______________________________________
Student evaluations of university professors
are of very limited validity. If, on the
evaluation form, the student were simply
asked to list their anticipated course
grade, an extremely high correlation between
the evaluation and the expected grade would
be observed.
This is well known and clearly invalidates
the entire exercise. The obvious result is
extreme grade inflation. It also results in
exorcising the best teachers from the
classroom -- the persons who refuse to
succumb to the resultant pressure for grade
inflation.
The pressure comes from the "Administrative
Class" that operates the universities. For
the administrator, a good teacher is, by
definition, someone who gives high grades.
Their logic is childishly simple. If the
students get poor grades, then the teacher
either cannot teach, or the teacher cannot
motivate the students to learn, or both.
-- Charles A. Weatherford, professor
and Chairman of Physics, Florida A&M
University (posted 1/12, 12:35 p.m.,
E.S.T.)
-----------------------------------------
There is currently a debate simmering (it's
not quite raging, yet) among health-care
providers and evaluators concerning
"consumer satisfaction" surveys. At times it
seems that, while "everybody's doing them,"
no one is really certain what they measure
or what their worth is in evaluating
providers (individuals or organizations) of
health care.
In many ways, this parallels the situation
in academe. It is important, for a number of
reasons, to allow consumers to provide an
assessment of their experiences -- in the
emergency room or the classroom. Such
assessments should be one element of an
"evaluation" of the service or experience.
To allow them to become the major type of
information -- or worse, the only
information -- to judge the worth or value
of the enterprise is tempting because
questionnaire-type surveys are so easy to
administer. But this focus on only one
source or type of information should be
resisted, if possible.
To quote a famous oracle of program
evaluation (or was it Yogi Berra?), we
should seek out simplicity and them distrust
it!
-- John Myers, Administrator of
Planning & Evaluation, Trumbull County
Alcohol, Drug, and Mental Health Board
(posted 1/16, 11:30 a.m., E.S.T.)
-------------------------------------------
The deleterious effects of student
evaluations are diverse and affect all
professors who aspire to maintain high
standards in their evaluation of student
performance. What I find particularly
distressing is the general attitude of many
students who "know the game." That is to
say, they are well aware of the impact of
their assessments and will often focus on
mundane aspects of a professor's performance
simply because they have been awarded a
grade which did not fit their expectations
before they entered the course.
If a professor is to believe or place any
credence to these, often not relevant,
comments she/he will in effect consciously
or subconsciously construct course
requirements based not on expertise gained
from present and past research, but; on what
is needed to satisfy the wishes of students
who can hardly be expected to promote a
course which requires "good old fashioned
hard work."
-- Al Figone, professor, Humboldt State
University (posted 1/12, 12:30 p.m.,
E.S.T.)
---------------------------------------
Modern man has developed a kind of Gallup-poll mentality,
relying on quantity instead of quality and yielding
to expediency.
>>Walter Gropius
Successful teachers are effective in spite of
the psychological theories they suffer under.
>>Old Educational proverb
>>
---------------------------------->>>
[[(#:-) >>>>
---------------------------------------->>>
+
....John C Damron, PhD *
....Douglas College, DLC * * *
....P.O Box 2503 * * *
....New Westminster, British Columbia
....Canada V3L 5B2 FAX: (604) 527-5969
....e-mail: [EMAIL PROTECTED]
http://www.douglas.bc.ca/
http://www.douglas.bc.ca/psychd/index.html
Student Ratings Critique:
http://www.mankato.msus.edu/dept/psych/Damron_politics.html
