If you object to cross posting, long posts (21kB), referencing,
unusual (but transparent) formatting, or consider this topic to be
irrelevant, please hit the DELETE button now! And if you reply to
this post, please don't hit the reply button unless you prune the
original message normally contained in your reply down to a few
lines, otherwise you may inflict this entire post yet again on
suffering list subscribers.
In my post "Re: Measuring Teaching Performance" [Hake (2005a)], I
wrote [bracketed by lines "HHHHHHHHHHH. . . .":
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
The above pre/post testing recommendation appears to conflict with
the opinion of psychologist David Berliner (2005). Berliner writes:
". . . .measurement of [teachers'] success in promoting learning
through 'pay for performance' or 'value-added' assessments is so
filled with psychometric problems that no current system is
acceptable for assessing this dimension of teacher quality."
But the evident success of physics education researchers in the use
of pre/post testing to gauge student learning and thereby improve
introductory courses [most notably at Harvard [Crouch & Mazur (2002)
and MIT (Dori & Belcher (2004)] seems to contradict Berliner's claim,
suggesting that psychometric problems may not be as serious [Hake
(2004), Scriven (2004)] as many believe. . .[see e.g, Suskie (2004)]
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
The above passage stimulated two responses (A & B below), both
countering my claim that the Berliner quote was contrary to the
apparent success of pre/post testing in physics education research:
A. Nathaniel Lasry (2005), in his PhysLrnR post of 6 Jun 2005, wrote
[bracketed by lines "LLLLLLLLLL. . . . ."; slightly edited]:
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL
Although it seems clear that pre-post testing is a good way to gauge the
quality of teaching I think there may be validity to Berliner's (2005) concern:
". . . . Measurement of [teachers'] success in promoting learning
through 'pay for performance' or 'value-added' assessments is so
filled with psychometric problems that no current system is
acceptable for assessing this dimension of teacher quality."
In Hake's (1998a,b) meta-analysis that showed an almost two-standard
deviation superiority of Interactive Engagement (IE) over Traditional
(T) methods of teaching Newtonian mechanics, no instructor pay scheme
was associated with normalized gains. What do you think would have
happened if this were not the case? It seems not too unlikely that
some traditional courses would "magically" increase in normalized
gain.
As Clement (2005) points out concerning the TX case: "Some
instructors will just ratchet up their grades [or FCI scores]. And
then the institutions will do the same type of cover up being done in
TX high schools. . . . After all if you don't know how to make
students successful, the only thing you can do is cheat."
This may prove to be particularly true when people's livelihood is at
stake. So essentially, as with learning, assessment measures need to
be contextualized. Hake's analysis of pre-post testing is
psychometrically sound but contextually different from Berliner's
concern...
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL
B. In a similar vein, Tom Pelton in his AERA-D post of 5 June 2005, wrote:
PPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP
There is no inconsistency or contradiction here. Berliner has correctly
identified the [fact?] that it is nearly impossible to test for
teacher quality.
The two situations you describe. . . [in the section bracketed by
lines "HHHHH. . . ." above] are entirely different.
The use of appropriate assessments by skilled teachers to support learning is
bound to be fruitful - as you and many others have found.
The inappropriate use of student assessments to monitor the quality
of teachers is bound to lead to many abuses (by both administrators
and teachers). These abuses (e.g., teaching to the test, gaming the
system, fraud, etc.) inevitably lead to the invalidation of what
could otherwise be effective assessment instruments.
PPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP
I agree with Lasry and Pelton that whether or not it's possible to
measure the *relative* effectiveness of a teacher in promoting
student learning (as by *normalized* gains) depends on the context of
the measurement, but disagree that there is validity to Berliner's
concerns **as he presented them in Berliner (2005)**.
As I indicated in Hake (2005c), according to "Campbell's Law"
[Campbell (1975), Nichols & Berliner (2005)]:
"The more any quantitative social indicator is used for social decision
making, the more subject it will be to corruption pressures and the
more apt it will be to distort and corrupt the social processes it is
intended to monitor."
*On the basis of "Campbell's Law"* one could argue that any attempt
to measure the relative effectiveness of a teacher in promoting
student learning *in the context of high-stakes testing* is doomed to
failure.
However, Berliner's argument that "no current system is acceptable
for assessing this dimension of teacher quality" doesn't appear to be
based on "Campbell's Law," but rather on "psychometric problems." I
assume Berliner [as others in the Psychology/Education/Psychometric
(PEP) community, e.g., Cronbach & Furby (1970), Suskie (2004)] would
claim that such "psychometric problems" exist even in the low-stakes
formative pre/post testing assessments of physics education research.
(These are formative in the sense that they are being used in an
ongoing process to improve physics instruction nation wide, and not
to arrive at a summative evaluation of individual course or
individual teacher merit.)
To put Berliner's ***quote*** in context, in "The Near Impossibility
of Testing for Teacher Quality," Berliner (2005) wrote:
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
The federal No Child Left Behind Act of 2001 (NCLB) mandated that a
highly qualified teacher be in all our nation's classrooms by
academic year 2005-2006. To accomplish this laudable goal, each state
must define what it means by a *highly qualified* teacher . . . .
examination of the federal requirement for highly qualified teachers
reveals that no means are offered to accomplish the desired goal,
suggesting that nothing is expected to happen. Moreover, the language
used to rally the politically faithful is kept purposely ambiguous,
with the term *highly qualified* providing no concrete referents for
anyone to understand what is so ardently being promoted . . . . It
should first be noted that successful teaching (evidence of student
learning) is not part of the assessment of beginning teachers. So
half of what it means to be highly qualified is ignored at the start
of one's career. Moreover, after teachers are more experienced,
***measurement of their success in promoting learning through 'pay
for performance' or 'value-added' assessments is so filled with
psychometric problems that no current system is acceptable for
assessing this dimension of teacher quality***.".
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Two questions:
1. Is "Campbell's Law" generally accepted among social scientists?
2. *Assuming* that "Campbell's Law" is generally correct, does it
condemn to failure the National Academy of Education's proposal [NAE
(2005), Hake (2005e)] to develop a "national performance-based
testing program for teachers that assesses the knowledge and skills
described [in the NAE report] through actual demonstration of
teaching practice."
Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<[EMAIL PROTECTED]>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>
"Teacher quality is determined less by a teacher's formal
qualifications and more by that teacher's ability to make
pedagogically fruitful use of materials, students' work, and their
own subject matter knowledge."
Cohen, Raudenbush, & Ball (2002)
REFERENCES
Berliner, D. 2005. "The Near Impossibility of Testing For Teacher
Quality," Journal of Teacher Education 56(3): 205-213, online free to
subscribers ($25 for non-subscribers) at
<http://jte.sagepub.com/content/vol56/issue3/>.I thank David Berliner
and Dewey Dykstra for sending me a copies.
Campbell, D. T. 1975. "Assessing the impact of planned social
change," in G. Lyons, ed., Social research and public policies: The
Dartmouth/OECD Conference, Chapter 1, pp. 3-45. Dartmouth College
Public Affairs Center, p. 35; online at Western Michigan University
as Occasional Paper #8 at
<http://www.wmich.edu/evalctr/pubs/ops/>, or download directly at
<http://www.wmich.edu/evalctr/pubs/ops/ops08.pdf> (196 kB).
Cohen, D.K., S.W. Raudenbush, & D.L. Ball. 2002. "Resources.
Instruction, & Research," in Mosteller & Boruch (2002).
Clement, J. 2005. "Re: The Archimedean Lever," PhysLrnR post of 6 Jun
2005 11:52:14-0500; online at
<http://listserv.boisestate.edu/cgi-bin/wa?A2=ind0506&L=physlrnr&F=&S=&X=3214761E9EC111058A&[EMAIL PROTECTED]&P=1390>.
The encyclopedic URL indicates that PhysLrnR is one of the few
discussion lists that denies archive access to non-subscribers :-( -
WHY ?? However, it takes only a few minutes to subscribe by following
the simple directions at
<http://listserv.boisestate.edu/archives/physlrnr.html> / "Join or
leave the list (or change settings)" where "/" means "click on." If
you're busy, then subscribe using the "NOMAIL" option under
"Miscellaneous." Then, as a subscriber, you may access the archives
and/or post messages at any time, while receiving NO MAIL from the
list!
Cronbach L. & L. Furby. 1970. "How we should measure 'change' - or
should we?" Psychological Bulletin 74: 68-80.
Crouch, C.H. & E. Mazur. 2001. "Peer Instruction: Ten years of
experience and results," Am. J. Phys. 69: 970-977; online at
<http://mazur-www.harvard.edu/library.php>, search "All Education
Areas" for author "Crouch" (without the quotes).
Dori, Y.J. & J. Belcher, J. 2004. "How Does Technology-Enabled Active
Learning Affect Undergraduate Students' Understanding of
Electromagnetism Concepts?" To appear in The Journal of the Learning
Sciences 14(2), online at
<http://web.mit.edu/jbelcher/www/TEALref/TEAL_Dori&Belcher_JLS_10_01_2
004.pdf> (1 MB).
Hake, R.R. 1998a. "Interactive-engagement vs traditional methods: A
six-thousand-student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66: 64-74; online as ref. 24 at
<http://www.physics.indiana.edu/~hake>, or simply click on
<http://www.physics.indiana.edu/~sdi/ajpv3i.pdf> (84 kB).
Hake, R.R. 1998b. "Interactive-engagement methods in introductory
mechanics courses," online as ref. 25 at
<http://www.physics.indiana.edu/~hake>, or simply click on
<http://www.physics.indiana.edu/~sdi/IEM-2b.pdf> (108 kB) - a
crucial companion paper to Hake (1998a).
Hake, R.R. 2004. "Re: pre-post testing in assessment," online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0408&L=pod&P=R9135&I=-3>.
Post of 19 Aug 2004 13:56:07-0700 to AERA-D, AERA-J, EDSTAT-L,
EVALTALK, PhysLrnR, and POD.
Hake, R.R. 2005a. "Re: Measuring Teaching Performance," online at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0505&L=aera-j&T=0&O=A&P=962>.
Post of 13/14 May 2005 to AERA-C, AERA-D, AERA-J, AERA-K, AERA-L,
ASSESS, Biopi-L, Chemed-L, EvalTalk, Math-Learn, Phys-L, PhysLrnR
Physhare, POD, & STLHE-L, & TeachingEdPsych. See also Hake (2005b,c).
Hake, R.R. 2005b. "Teachers: the Archimedean Lever for Elevating
Public Schools," online at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0506&L=aera-l&T=0&F=&S=&P=160>.
Post of 3 Jun 2005 17:18:11-0700 to AERA-C, AERA-G, AERA-GSL, AERA-H,
AERA-J, AERA-K, AERA-L, AP-physics, ASSESS, Biopi-L, Chemed-L,
EvalTalk, Math-Learn, Phys-L, PhysLrnR, Physhare, POD, STLHE-L,
TeachingEdPsych, and TIPS.
Hake, R.R. 2005c. "The Archmedean Lever," online at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0506&L=aera-l&T=0&F=&S=&P=375>.
Post of 4 Jun 2005 15:54:40-0700 to AERA-C, AERA-D, AERA-G,
AERA-GSL, AERA-H, AERA-J, AERA-K, AERA-L, AP-physics, ASSESS,
Biopi-L, Chemed-L, EdStat-L, EvalTalk, Math-Learn, Phys-L, PhysLrnR,
Physhare, POD, STLHE-L, TeachingEdPsych, & TIPS.
Hake, R.R. 2005d. "Will the No Child Left Behind Act Promote Direct
Instruction of Science?" Am. Phys. Soc. 50: 851 (2005); APS March
Meeting, Los Angles, CA. 21-25 March; online as ref. 36 at
<http://www.physics.indiana.edu/~hake>, or download directly by clicking on
<http://www.physics.indiana.edu/~hake/WillNCLBPromoteDSI-3.pdf> (256
kB).
Hake, R.R. 2005e. "Panel Urges New Testing For Teachers," AERA-L post
of 3 Jun 2005 15:48:0-0700; online at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0506&L=aera-l&T=0&F=&S=&P=52>.
Lasry, N. 2005. "Re: The Archimedean Lever," PhysLrnR post of 6 Jun
2005 11:01:26-0700; online at
<http://listserv.boisestate.edu/cgi-bin/wa?A2=ind0506&L=physlrnr&F=&S=&X=3DBDF931B753231734&[EMAIL PROTECTED]&P=1286>.
For a comment on PhysLrnR's pathologic URL see Clement (2005).
Mosteller, F. & R. Boruch, eds. 2002. "Evidence Matters: Randomized
Trials in Education Research". Brookings Institution.
NAE. 2005. National Academy of Education <http://www.nae.nyu.edu/>,
Introduction and Policy Recommendations (Chapter 4) of "A Good
Teacher in Every Classroom" are online at
<http://www.nae.nyu.edu/Introduction&Recommendations.pdf> (96 kB).
Nichols, S.L & D.C. Berliner. 2005. "The Inevitable Corruption of
Indicators and Educators Through High-Stakes Testing," Arizona State
Univ. Education Policy Studies Laboratory, online at
<http:www.asu.edu/educ/epsl/EPRU/documents/EPSL-0503-101-EPRU.pdf> (1.7 MB).
Pelton, T. "Re: The Archimedean Lever," AERA-D post of 5 Jun 2005
01:10:12 -0700; online at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0506&L=aera-d&T=0&F=&S=&P=3718>.
Scriven, M. 2004. "Re: pre- post testing in assessment," AERA-D post
of 15 Sep 2004 19:27:14-0400; online at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0409&L=aera-d&T=0&F=&S=&P=1952>.
Suskie, L. 2004. "Re: pre- post testing in assessment," ASSESS post
19 Aug 2004 08:19:53-0400; online at
<http://lsv.uky.edu/cgi-bin/wa.exe?A2=ind0408&L=assess&D=0&T=0&P=7492&F=P>.
--
Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<[EMAIL PROTECTED]>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>
---
You are currently subscribed to tips as: [email protected]
To unsubscribe send a blank email to [EMAIL PROTECTED]
