My experience with randomizing the sequence of items vs. leaving them unscrambled is similar to Karl's. FWIW, back in the 1980s when I worked with subject matter experts to construct civil service entry level and promotional exams for the city of New York, the policy, as I recall it, was to randomly scramble all of the questions and each question's options (we were not permitted to use all or none of the above).
I also found the following article, which confirms the sequence effect. Item order affects performance on multiple - choice exams Authors: Balch, William R. , Pennsylvania State U, Altoona, PA, US Source: Handbook for teaching introductory psychology: Vol. 3: With an emphasis on assessment. Griggs, Richard A. (Ed. ); pp. 106-108. Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers, 2002. ix, 257 pp. ISBN: 0-8058-3921-6 (Paperback) Language: English Keywords: test item order; sequential order; chapter contiguity; random order; exam performance; multiple choice final exam; general psychology courses Abstract: This reprinted article originally appeared in Teaching of Psychology , 1989(Apr), Vol 16(2), 75–77. (The following abstract of the original article appeared in record 1990-06169-001. ) 404 general psychology students were assigned to 1 of 3 different item orders (sequential, chapter contiguity, and random) of the same final exam consisting of 75 multiple - choice questions. In the sequential order exam, items appeared in the same sequence in which their supporting material was presented in the textbook and lectures. For the chapter contiguity order exam, items based on the same chapter appeared together, but were not sequentially arranged within or between chapters. The order of the 3rd exam was random. Scores for the sequential order exam were higher than for the other two. There were no significant differences in the completion times for any of the exams. (PsycINFO Database Record (c) 2011 APA, all rights reserved) Miguel ----- Original Message ----- From: "Stuart McKelvie" <[email protected]> To: "Teaching in the Psychological Sciences (TIPS)" <[email protected]> Sent: Saturday, May 5, 2012 7:31:36 PM Subject: RE: [tips] Curious about exam construction Dear Tipsters, Karl wrote: I once, in a large class and with multiple forms of the exam (same questions, different order), mistaking used one form that had the questions appearing in order of the page in the text on the topic was presented. Performance on that form was about a letter grade higher than on the form with items scrambled. Comment: I assume that the implication here is that the higher letter grade is spurious. If that is the implication, I wonder why. That is, why is it problematic for students to obtain more correct answers when the questions are in the same sequence as the topics in the textbook? To put this another way, why is randomizing questions a good thing? My hunch is that if students benefit by seeing connections among questions, then that is a good thing! Sincerely, Stuart ______________________________ “ Rectu Cultus Pectora Roborant ” Stuart J. McKelvie , Ph.D., Department of Psychology, Bishop’s University, 2600 rue College, Sherbrooke (Borough of Lennoxville), QC J1M 1Z7, Canada. “ Floreat Labore ” ______________________________ --- You are currently subscribed to tips as: [email protected] . To unsubscribe click here: http://fsulist.frostburg.edu/u?id=466839.0421d1005414eed82340aa280e7ce629&n=T&l=tips&o=17647 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-17647-466839.0421d1005414eed82340aa280e7ce...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=17661 or send a blank email to leave-17661-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
