ABSTRACT: I respond to two questions from PsychTeachers on how they
might gauge their teaching effectiveness. I contrast *indirect*
methods such as (1) Reformed Teaching Observation Protocol (RTOP),
(2) Student Evaluations Of Teaching (SET's), (3) Course Exams or
Final Grades, (4) National Survey Of Student Engagement (NSSE), and
(5) Student Assessment Of Learning Gains" (SALG); with a *direct*
method of pre/post testing using (a) valid and consistently reliable
tests devised by disciplinary experts, and (b) traditional courses as
controls. I indicate why the apparent failure of psychologists to
research the effectiveness of their own introductory courses as an
important issue in education research.
Those who dislike long posts (19 kB), references, or cross-posting;
or who have no interest in gauging how they're doing, are urged to
hit the DELETE button. And if you reply PLEASE DON'T HIT THE REPLY
BUTTON UNLESS YOU PRUNE THE COPY OF THIS POST THAT MAY APPEAR IN YOUR
REPLY DOWN TO A FEW RELEVANT LINES, OTHERWISE THE ENTIRE POST MAY BE
NEEDLESSLY RESENT TO SUBSCRIBERS.
In his PsychTeacher post of 7 Sep 2005 titled "Re: How do you gauge
how you're doing?" Jessey Bernstein (2005) wrote [bracketed by lines
"BBBBB. . . ."; slightly edited; my CAPS]:
BBBBBBBBBBBBBBBBBBBBBBBBBBB
As my first week of teaching unrolls, I started to wonder about how I
(and you all) know if we are doing a good job as teachers. I am
thinking about it on various levels. For example, how do you know if:
(a) you gave a good lecture (no one sleeping or whispering to
neighbor or rolling their eyes? People asking thoughtful questions?);
and on a larger scale,
(b) how do you know if you are teaching well? (Exam performance?
Assignment quality? Course evals - YUCK!)
IT SEEMS SO IMPORTANT TO KNOW IF WE ARE EFFECTIVE, YET I AM ACTUALLY
NOT SURE HOW TO GAUGE IT.
What do you all use as feedback?
BBBBBBBBBBBBBBBBBBBBBBBBBBB
Great minds run in the same direction. In a PsychTeacher post of 6
May 2005 titled "Re: Measuring Teaching Performance," Jessie Owen
wrote:
"I was wondering what, if any, evaluations have you used to measure
what professors are doing in class? Please note I am not interested
in measures that only examine student satisfaction or preference
(although these items on a measure would be helpful). I have looked
in the literature and have found some useful starting places;
however, I figured that this list would a great resource to help
generate some ideas and some insights about the practicality of these
measures."
Jessie's request garnered zero responses from PsychTeachers, but I
responded in two posts [Hake (2005a,b)] to various discussion lists
other than PsychTeacher - from which I'm banned :-( . Both posts may
be scanned by simply clicking on the URL's indicated in the
references.
I assume those responses to Jessie Owen will at least partially
suffice as responses to the very similar post by Jessey Bernstein.
Jessey's post stimulated posts by:
(a) Elizabeth Seebach, who suggested a variant of the famous "Minute
Papers" invented by Berkeley physicist Charles Schwartz (1983) [see
also (Hake 2001)],
(b) Missa Murry Eaton, who suggested that "a combination of early
informal feedback, observation by more senior instructors, and the
(dreaded) end-of-semester evaluations [are] good starting place[s]."
(c) Martha Zlokovich, who had no answers for Jessey, but raised many
good questions about measuring teaching effectiveness and the use of
Student Evaluations of Teaching . . .[in my opinion these are NOT
valid measures of the cognitive (as opposed to the affective) impact
of courses - see Hake (2002)].
IF teaching effectiveness is to be gauged by *student learning*, then
there are several methods that might be considered other than those
suggested by Seebach and Eaton. The abstract of "Re: Measuring
Teaching Performance" [Hake (2005b)] reads as follows [bracketed by
lines "HHHHHH. . . ."; see that post for references to the measures
mentioned]:
HHHHHHHHHHHHHHHHHHHHHHHHHHH
Assuming that "teaching performance" is gauged by *student learning*
and not *teacher behavior*. . . [per se]. . , I discuss: five
INDIRECT and therefore *problematic* measures of teaching
performance: (1) Reformed Teaching Observation Protocol (RTOP), (2)
Student Evaluations Of Teaching (SET's), (3) Course Exams or Final
Grades, (4) National Survey Of Student Engagement (NSSE), and (5)
Student Assessment Of Learning Gains" (SALG). These are contrasted
with a DIRECT measure of teaching performance (i.e., student
learning) pioneered by physics education researchers: pre/post
testing with valid and consistently reliable diagnostic tests based
on thorough qualitative and quantitative research by disciplinary
experts. . .[using the traditional courses as controls].
HHHHHHHHHHHHHHHHHHHHHHHHHHH
Pre/post testing is being done more and more in Astronomy, Economics,
Biology, Chemistry, Computer Science, and Engineering, as referenced
in Hake (2004a,b)], but NOT psychology (WHY ??).
Doubtless some psychologists may wonder why non-psychologists should
care whether or not psychologists research the effectiveness of their
own courses? In Hake (2005d) I wrote [see that post for references
other than Hake (2002; 2004c,d; 2005e,f) & Kelly (2003)]:
HHHHHHHHHHHHHHHHHHHHHHHHHH
I regard the apparent failure of psychologists to research the
effectiveness of their own introductory courses as an important issue
in education research because, among other things:
(1) One might expect psychologists with their long history of
education research [Berliner (1993), Lagemann (2000)] and their
leading role in classroom-oriented "Design Based Research" [Kelly
(2003); Hake (2004c,d; 2005e)] to be in the vanguard of those
actively researching the effectiveness of own courses and thus
serving as role models for other faculty.
(2) Educational psychologists often staff the "Teaching and Learning
Centers" of U.S. universities and thus might (but generally do not)
influence faculty to research the cognitive effectiveness of their
courses through valid and consistently reliable diagnostic tests
developed by disciplinary experts, rather than through the usual
problematic [Hake (2002)] student evaluations.
(3) Psychologists and psychometricians seem to be in control of the
U.S. Dept. of Education's "What Works Clearinghouse"
<http://www.w-w-c.org/> and NCLB testing of "science achievement" to
commence in 2007 [see Hake (2005f)]. Why should they be the arbiters
of "What Works," when, as far as I know, they haven't even bothered
to research "What Works" in their own courses?
HHHHHHHHHHHHHHHHHHHHHHHHHH
BTW, for a recent debate:
Resolved - PSYCHOLOGISTS *DO* RESEARCH THE EFFECTIVENESS OF THEIR OWN
INTRODUCTORY COURSES: Pro: R.J. Sternberg, Con: R.R. Hake,
see Hake (2005g,h).
Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<[EMAIL PROTECTED]>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>
People have nowadays . . . got a strange opinion that everything
should be taught by lectures. Now, I cannot see that lectures can do
so much good as reading the books from which the lectures are taken .
. .Lectures were once useful; but now, when we can all read, and
books are so numerous, lectures are unnecessary.
Samuel Johnson according to James Boswell's "Life of Samuel Johnson, LL.D.,
Encyclopedia Britannica (1952).
REFERENCES
Bernstein, J. 2005. "How do you gauge how you're doing?" PsychTeacher
post of 7 Sep 2005 18:47:28-0700; online at
<http://list.kennesaw.edu/cgi-bin/web-admin?A2=ind0509&L=psychteacher&O=A&X=58581F304BBF53F5F2&[EMAIL
PROTECTED]&P=10177>.
Hake, R.R. 2001. "Schwartz Invented Minute Papers," online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0105&L=pod&P=R4417&I=-3>.
Post of 5 May 2001 19:59:40-0700 to AERA-J, Phys-L, PhysLrnR,
Physhare, POD, & STLHE-L.
Hake, R.R. 2002. "Re: Problems with Student Evaluations: Is
Assessment the Remedy?" online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0204&L=pod&P=R14535>. Post
of 25 Apr 2002 16:54:24-0700 to AERA-D, ASSESS, EvalTalk, Phys-L,
PhysLrnR, POD, & STLHE-L. Slightly edited and improved on 16 November
2002 as ref. 18 at <http://www.physics.indiana.edu/~hake> or download
directly at <http://www.physics.indiana.edu/~hake/AssessTheRem1.pdf>
(72 kB)[This is the best version.]. Also online in HTML at
<http://www.stu.ca/~hunt/hake.htm> as one of the many resources in
Russ Hunt's annotated bibliography of articles and books on student
evaluation of teaching <http://www.stu.ca/~hunt/evalbib.htm>.
Hake, R.R. 2004a. "Re: Measuring Content Knowledge", online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0403&L=pod&P=R13279&I=-3>.
Post of 14 Mar 2004 16:29:47-0800 to ASSESS, Biopi-L, Chemed-L,
EvalTalk, Physhare, Phys-L, PhysLnrR, POD, and STLHE-L; later sent to
AERA-D with a few corrections where it appears at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0403&L=aera-d&P=R3625&I=-3>.
Hake, R.R. 2004b. "Re: Measuring Content Knowledge", online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0403&L=pod&O=A&P=17167>.
Post of 15 Mar 2004 14:29:59-0800 to ASSESS, EvalTalk, Phys-L,
PhysLrnR, & POD.
Hake, R.R. 2004c. "Design-Based Research: A Primer for Physics
Education Researchers," submitted to the "American Journal of
Physics" on 10 June 2004; online as reference 34 at
<http://www.physics.indiana.edu/~hake>, or download directly by
clicking on <http://www.physics.indiana.edu/~hake/DBR-AJP-6.pdf>
(310kB).
Hake, R.R. 2004d. "Design-Based Research: Old PER Wine in a New
Bottle," submitted to PERC 2004 on 16 July 2004; online as reference
35 at <http://www.physics.indiana.edu/~hake>, or download directly as
a 120 kB pdf by clicking on
<http://www.physics.indiana.edu/~hake/PERC2004-Hake6.pdf>. PER =
Physics Education Research.
Hake, R.R. 2005a. "Re: Measuring Teaching Performance," online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0505&L=pod&P=R6968&I=-3 >.
Post of 11 May 2005 18:30:15-0400 to ASSESS, EVALTALK, PhysLrnR, POD,
TIPS, & TeachingEdPsych. Stimulated by Jessie Owen's PsychTeacher
post of 6 May 2005.
Hake, R.R. 2005b. "Re: Measuring Teaching Performance," online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0505&L=pod&P=R9303&I=-3>.
Post of 13 May 2005 17:16:48-0400 to AERA-D, AERA-J, AERA-K, AERA-L,
ASSESS, BIOPI-L, CHEMED-L, EVALTALK, Math-Learn, PHYS-L, Physhare,
PhysLrnR, POD, STLHE-L, & TIPS. Stimulated by Jessie Owen's
PsychTeacher post of 6 May 2005.
Hake, R.R. 2005c. "Re: Why Don't Psychologists Research the
Effectiveness of Their Own Introductory Courses?"; online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0501&L=pod&P=R10628>. Post
of 18 Jan 2005 20:57:21-0800 to AERA-D, EdStat-L, EvalTalk, PhysLrnR,
PsychTeacher, & TeachingEdPsch.
Hake, R.R. 2005d. "Do Psychologists Research The Effectiveness Of
Their Own Introductory Courses?" TIPS post of 19 Feb 2005
07:58:43-0800; online at
<http://www.mail-archive.com/[email protected]/msg13133.html>.
See also Hake (2005c).
Hake, R.R. 2005e. "Design-Based Research in Physics Education
Research: A Review," in "Design Experiments in Education," A.E. Kelly
& R.A. Lesh, eds., Erlbaum, to be published.
Hake, R.R. 2005f. "Will the No Child Left Behind Act Promote Direct
Instruction of Science?" Am. Phys. Soc. 50: 851 (2005); APS March
Meeting, Los Angles, CA. 21-25 March; online as ref. 36 at
<http://www.physics.indiana.edu/~hake>, or download directly by
clicking on
<http://www.physics.indiana.edu/~hake/WillNCLBPromoteDSI-3.pdf> (256
kB).
Hake, R.R. 2005g. "Do Psychologists Research the Effectiveness of
Their Courses? Sternberg Says YES," online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0507&L=pod&P=R11834&I=-3>,
Post of 21 Jul 2005 18:22:52-0700 to AERA-C, AERA-D, AERA-J AERA-L,
ASSESS, EvalTalk, PhysLnrR, POD, & STLHE-L.
Hake, R.R. 2005h. "Do Psychologists Research the Effectiveness of
Their Courses? Hake Responds to Sternberg," online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0507&L=pod&P=R11939&I=-3>.
Post of 21 Jul 2005 22:55:31-0700 to AERA-C, AERA-D, AERA-J, AERA-L,
ASSESS, EvalTalk, PhysLnrR, POD, & STLHE-L.
Kelly, A.E. 2003. "Research as Design," Educational Researcher 32(1):
3-4; online at <http://www.aera.net/publications/?id=393>. Introduces
an issue devoted to the theme "The Role of Design in Educational
Research."
Owen, J. 2005. Re: Measuring Teaching Performance PsychTeacher post
of 6 May 2005 07:48:20-0700; online at
<http://list.kennesaw.edu/cgi-bin/web-admin?A2=ind0505&L=psychteacher&O=A&X=742EAA2089845DB664&[EMAIL
PROTECTED]&P=3233>.
Schwartz, C. 1983. "Minute Papers" as described in "ABC's of teaching
with excellence: A Berkeley compendium of suggestions for teaching
with excellence," B.G. Davis, L. Wood, R.C. Wilson; online at
<http://teaching.berkeley.edu/compendium/>. The Minute Paper
description is online at
<http://teaching.berkeley.edu/compendium/suggestions/file95.html>.
---
You are currently subscribed to tips as: [email protected]
To unsubscribe send a blank email to [EMAIL PROTECTED]
