If you reply to this long (19 kB) post please don't hit the reply button unless you prune copy of this post that may appear in your reply down to a few relevant lines, otherwise the entire already archived post may be needlessly resent to subscribers.
*********************************************** ABSTRACT: Gina Hiatt (2008) of the POD list has called attention to Kevin Carey's (2008) provocative "Inside Higher Ed" report "Where's the Data?" Carey wrote: ". . . .when it comes to the central enterprise of higher education - teaching students - we don't know if the reigning professional qualification system works, or how many professors we actually need. And this is true for all kinds of other basic elements of college teaching and learning - curricula, training, pedagogy, and much more. . . . . Why doesn't anyone ever study how much learning varies between [courses], and why?" Carey, along with most of academia, appears to be either unaware or dismissive of the fact that formative pre/post testing is being successfully employed to enhance student learning in many science, math, and engineering, and economics (but not psychology!) courses. For introductory physics courses it's been found that pre-to-post-test average normalized learning gains for "interactive engagement" courses are about two standard deviations greater than those for traditional passive-student courses. *********************************************** Kevin Carey, research and policy manager of the "Education Sector" <http://www.educationsector.org/> and a frequent contributor <http://www.insidehighered.com/views/carey> to "Inside Higher Ed", wrote [bracketed by lines "CCCCCC. . . . .";]: CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC . . . . . when it comes to the central enterprise of higher education - teaching students - we don't know if the reigning professional qualification system works, or how many professors we actually need. And this is true for all kinds of other basic elements of college teaching and learning - curricula, training, pedagogy, and much more. . . . . . Transcript studies . . . .[ Adelman (2004)]. . . .indicate that 20 percent of all course credits earned by college graduates come in just 13 introductory courses like English Composition, Calculus, and Introduction to Economics. Seventy-one percent of all college graduates take some version of Psychology 101. Calculus is pretty much calculus wherever you go (or should be). And even in cases where curricula vary between institutions, larger universities routinely teach many sections of the same course every semester. Why doesn't anyone ever study how much learning varies between them, and why? CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC Carey is evidently either unaware or dismissive of fact ever since the ground-breaking research of Halloun & Hestenes (1985a,b), physics education researchers have been studying how much student learning varies between various types of introductory physics courses [see e.g., Stokstad (2001)], but the "Lessons of the Physics Education Reform Effort" [Hake (2002a, 2007b)] have been largely ignored by academia, obsessed with its misconception that Student Evaluations are valid gauges of the cognitive impact of courses - see e.g., "Re: Problems with Student Evaluations: Is Assessment the Remedy?"[Hake (2002b)]. Among those lessons are [Hake (2007b) - SEE THAT ARTICLE FOR THE REFERENCES other than Hake (2002a; 2005a; 2007a; 2008a,b); Halloun and Hestenes (1985a,b); Hestenes et al. (1992); Wieman & Perkins (2005); and Wilson & Bertenthal (2005).] Lesson 1: THE USE OF 'INTERACTIVE ENGAGEMENT' (IE) STRATEGIES CAN INCREASE THE EFFECTIVENESS OF CONCEPTUALLY DIFFICULT COURSES WELL BEYOND THAT OBTAINED BY TRADITIONAL (T) METHODS. . . . . . Education research in chemistry [Krause et al. (2004)]; engineering [Froyd et al. (2006), Evans et al. (2003)]; and introductory science education generally [Handelsman et al. (2004)], although neither as extensive nor as systematic as that in physics [McDermott and Redish (1999); Redish (1999); Thacker (2003); Heron & Meltzer (2005); Hake (1998a,b; 2002a,b; 2005a; 2006a,b;2007a,b); Wieman & Perkins (2005); Wieman (2005)] is consistent with the latter in suggesting that, in conceptually difficult areas, Interactive Engagement (IE) methods are more effective than traditional T passive-student methods in enhancing students' understanding. . . . .[ For introductory physics courses it's been found that pre-to-post-test average normalized learning gains for "interactive engagement" courses are about two standard deviations greater than those for traditional passive-student courses - for references see "Design-Based Research in Physics Education Research: A Review" [Hake (2008a)]. . . . . Furthermore, there is some preliminary evidence that learning in IE physics courses is substantially retained 1 to 3 years after the courses have ended [Chabay (1997), Francis et al. (1998), Bernhard (2001)]. I see no reason to doubt that enhanced understanding and retention would result from greater use of IE methods in other science, and even non-science, areas, but substantive research on this issue is sorely needed - see e.g., "The Physics Education Reform Effort: A Possible Model for Higher Education?" [Hake (2005a)]. Pre/post testing in biology [Klymkowsky et al. (2003), Klymkowsky (2007)]; and mathematics [Epstein (2005)] is just getting started; while pre/post test results in astronomy (Brogt et al. (2007) and geoscience [Libarkin & Anderson (2005)], have not, at this early stage, shown clear cut correlations between pre-to-posttest gain and pedagogical method, as has been shown in physics. Lesson 3: "HIGH-QUALITY STANDARDIZED TESTS OF THE COGNITIVE AND AFFECTIVE IMPACT OF COURSES ARE ESSENTIAL TO GAUGE THE RELATIVE EFFECTIVENESS OF NON-TRADITIONAL EDUCATIONAL METHODS. . . . . So great is the inertia of the educational establishment that three decades of physics education research [McDermott and Redish (1999)] demonstrating the futility of the passive-student lecture in introductory courses was ignored until Halloun and Hestenes (1985a,b) devised the Mechanics Diagnostic (MD) test of conceptual understanding of Newtonian mechanics. Among many other virtues, the MD and the subsequent Force Concept Inventory (FCI) (Hestenes et al. 1992) tests have two major advantages: (a) the multiple-choice format facilitates relatively easy administration of the tests to thousands of students; (b) the questions probe for a conceptual understanding of the basic concepts of Newtonian mechanics in a way that is understandable to the novice who has never taken a physics course, yet at the same time are rigorous enough for the initiate. Thus the questions can be given as an introductory course pretest in pre/post tests to directly determine course-induced gain in conceptual understanding. . . .[for a recent listing of "Formative Pre/post Tests For Various Disciplines" see Hake (2008b) - "formative" is used here in the sense defined by JCSEE (1994): "Formative evaluation is evaluation designed and used to improve an object, especially when it is still being developed." ]. . . . . In my opinion such DIRECT gain measurements of higher-order student learning are far superior to the INDIRECT (and therefore in my view problematic) gauges have been developed; e.g., Reformed Teaching Observation Protocol (RTOP), National Survey Of Student Engagement (NSSE), Student Assessment of Learning Gains (SALG), and Knowledge Surveys (KS's) [Nuhfer & Knipp (2003)]. For a discussion and references for all but the last see Hake (2005b.) BUT WAIT! 1. Can multiple choice tests gauge higher level cognitive outcomes such as the conceptual understanding of Newtonian mechanics? Wilson & Bertenthal (2005) think so, writing (p. 94): "Performance assessment is an approach that offers great potential for assessing complex thinking and learning abilities, but multiple choice items also have their strengths. For example, although many people recognize that multiple-choice items are an efficient and effective way of determining how well students have acquired basic content knowledge, many do not recognize that they can also be used to measure complex cognitive processes. For example, the Force Concept Inventory . . . [Hestenes, Wells, & Swackhamer, 1992] . . . is an assessment that uses multiple-choice items to tap into higher-level cognitive processes." 2. Considering the canonical arguments regarding the invalidity of pre/post testing evidence, should not all pre-to-post test gains cited above be viewed with grave suspicion? The dour appraisal of pre/post testing by Cronbach & Furby (1970) has echoed down though the literature to present day texts on assessment such as that by Suskie (2004)]. In my opinion, such pre/post paranoia and its attendant rejection of pre/post testing in evaluation, as used so successfully in physics education reform [McDermott and Redish (1999); Redish (1999); Thacker (2003); Heron & Meltzer (2005); Wieman & Perkins (2005), Wieman (2005)] is one reason for the glacial progress of educational research [Lagemann (2000)] and reform [Bok (2005)]. Fortunately formative pre/post testing is gradually gaining a foothold in undergraduate astronomy, biology, chemistry, economics, geoscience, and engineering, in addition to physics. For references see Hake (2004, 2007c, 2007d). Richard Hake, Emeritus Professor of Physics, Indiana University 24245 Hatteras Street, Woodland Hills, CA 91367 Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands. <[EMAIL PROTECTED]> <http://www.physics.indiana.edu/~hake/> <http://www.physics.indiana.edu/~sdi/> "The academic area is one of the most difficult areas to change in our society. We continue to use the same methods of instruction, particularly lectures, that have been used for hundreds of years. Little scientific research is done to test new approaches, and little systematic attention is given to the development of new methods. Universities that study many aspects of the world ignore the educational function in which they are engaging and from which a large part of their revenues are earned." - Richard M. Cyert, former president of Carnegie Mellon Univ. in "Problem Solving and Education: Issues in Teaching and Research," ed. by D.T. Tuma and F. Reif (Lawrence Erlbaum, 1980). REFERENCES [Tiny URL's courtesy <http://tinyurl.com/create.php>.] Adelman, C. 2004."The Empirical Curriculum: Changes in Postsecondary Course-Taking, 1972- 2000". U.S. Department of Education; online at <http://www.ed.gov/rschstat/research/pubs/empircurr/empircurric.pdf> (436 kB). Carey, K. 2008. "Where's the Data?" Inside Higher Ed., 8 August; online at <http://www.insidehighered.com/views/2008/08/08/carey>. Readers may wish to add to the profusion of comments on Carey's piece. Hake, R.R. 2002a. "Lessons from the physics education reform effort," Ecology and Society 5(2): 28; online at <http://www.ecologyandsociety.org/vol5/iss2/art28/>. Ecology and Society (formerly Conservation Ecology) is a free online "peer-reviewed journal of integrative science and fundamental policy research" with about 11,000 subscribers in about 108 countries. For an update on six of the fourteen lessons see Hake (2007b). Hake, R.R. 2002b. "Re: Problems with Student Evaluations: Is Assessment the Remedy?" online at <http://www.physics.indiana.edu/~hake/AssessTheRem1.pdf> (72 kB). Hake, R.R. 2003. "Re: A taxonomy," POD posts of 9 & 12 Jul 2003, online at <http://listserv.nd.edu/cgi-bin/wa?A2=ind0307&L=pod&P=R4226&I=-3> (a diagram is shown), and <http://listserv.nd.edu/cgi-bin/wa?A2=ind0307&L=pod&P=R5361&I=-3>. Hake, R. R. 2005a. "The Physics Education Reform Effort: A Possible Model for Higher Education," online at <http://www.physics.indiana.edu/~hake/NTLF42.pdf> (100 kB). This is a slightly edited version of an article that was (a) published in the National Teaching and Learning Forum 15(1), December 2005, online to subscribers at <http://www.ntlf.com/FTPSite/issues/v15n1/physics.htm> - if your institution doesn't subscribe, then it should, and (b) disseminated by the Tomorrow's Professor list <http://ctl.stanford.edu/Tomprof/postings.html> as Msg. 698 on 14 Feb 2006. Hake, R.R. 2007a. "Should We Measure Change? Yes!" online at <http://www.physics.indiana.edu/~hake/MeasChangeS.pdf> (2.5 MB), or as ref. 43 at <http://www.physics.indiana.edu/~hake>. To appear as a chapter in "Evaluation of Teaching and Student Learning in Higher Education" a Monograph of the American Evaluation Association <http://www.eval.org/>. Hake, R.R. 2007b. "Six Lessons From the Physics Education Reform Effort," Latin American Journal of Physics Education 1(1), September; online at <http://journal.lapen.org.mx/sep07/HAKE%20Final.pdf> (124 kB). Hake, R.R. 2008a. "Design-Based Research in Physics Education Research: A Review," in "Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching" [Kelly, Lesh, & Baek (2008)] - publisher's information at <http://tinyurl.com/4eazqs>; a pre-publication version of Hake's chapter is online at <http://www.physics.indiana.edu/~hake/DBR-Physics3.pdf> (1.1 MB). Hake, R.R. 2008b. "Formative Pre/post Tests For Various Disciplines," online as it appears on the OPEN AERA-D archives at <http://tinyurl.com/6cyemf> where URL's in my post are properly HOT-LINKED (unlike those on EdResMeth, EdStat, EvalTalk, and POD). Post of 7 and 8 July to AERA-D, ASSESS, Biopi-L, Chemed-L, DrEd, EdResMeth, EdStat, EvalTalk, IFETS, Net-Gold, PhysLrnR, POD, RUME, & WBTOLL-L. Halloun, I. & D. Hestenes. 1985a. "The initial knowledge state of college physics students." Am. J. Phys. 53: 1043-1055; online at <http://modeling.asu.edu/R&E/Research.html>. The print version contains the Mechanics Diagnostic test, precursor to the Force Concept Inventory [Hestenes et al. (1992)]. Halloun, I. & D. Hestenes. 1985b. "Common sense concepts about motion," Am. J. Phys. 53: 1056-1065; online at <http://modeling.asu.edu/R&E/Research.html>. Hestenes, D., M. Wells, & G. Swackhamer. 1992. "Force Concept Inventory," Phys. Teach. 30(3): 141-158, March; online (except for the test itself) at <http://modeling.asu.edu/R&E/Research.html>. The 1995 revision by Halloun, Hake, Mosca, & Hestenes is online (password protected) at the same URL, and is currently available in 15 languages: Chinese, Czech, English, Finnish, German, Greek, Italian, Malaysian, Persian, Portuguese, Russian, Slovak, Spanish, Swedish, & Turkish. A French version should soon be available. Hiatt, G. 2008. "Article from Inside Higher Ed," POD post of 9 Aug 2008 14:26:26 -0400; online at <http://tinyurl.com/5m3nqn>. JCSEE. 1994. Joint Committee on Standards for Educational Evaluation, The Program Evaluation Standards, 2nd ed., Sage. A glossary of evaluation terms from this publication is online at <http://ec.wmich.edu/glossary/prog-glossary.htf>. For a four-quadrant delineation of the formative/summative and public/private dimensions of assessment/evaluation see Hake (2003). Stokstad, E. 2001. "Reintroducing the Intro Course," Science 293: 1608-1610, 31 August; online at <http://tinyurl.com/266973>. Stokstad wrote: "Physicists are out in front in measuring how well students learn the basics, as science educators incorporate hands-on activities in hopes of making the introductory course a beginning rather than a finale." Wieman, C. & K. Perkins. 2005. "Transforming Physics Education," Phys. Today 58(11): 36-41; online as a 292 kB pdf at <http://tinyurl.com/4py56v>. [Wieman is a 2001 Physics Nobelist.] Wilson, M.R. & M.W. Bertenthal, eds. 2005. "Systems for State Science Assessment," Nat. Acad. Press; online at <http://www.nap.edu/catalog.php?record_id=11312>. --- To make changes to your subscription contact: Bill Southerly ([EMAIL PROTECTED])
