On Wed, 01 Mar 2006 09:50:40 -0800, Annette Taylor, Ph. D. wrote: > >I found the reference I was thinking of: > >Katz, S., Lautenschlager, G. J., Blackburn, A. B. & Harris, F. H. >(1990). Answering reading comprehension items without passages >on the SAT. Psychological Science, 1, 122-127. > >Here is the abstract: >Examined whether the Reading Comprehension (RC) task on the >Scholastic Aptitude Test (SAT) measures factors that are unrelated >to RC. Two experiments were conducted with 197 college students. >Performance of Ss on the RC task was well above chance when >reading passages were deleted. Moreover, Ss and test items >performed similarly with or without the passages: Individual performance >correlated with verbal SAT score, and the difficulty of items belonging >to a passage correlated with a normative measure based on equated delta. >Thus, performance on the RC task appears to depend substantially on >factors having nothing to do with understanding the passages normally >accompanying test questions. (PsycINFO Database Record (c) 2005 >APA, all rights reserved)
I don't think that this research really is relevant to the issue of bad multiple choice items/tests (which I interpret as reflecting questions and answers that are vague, ambiguous, are consistent with multiple interpretations, etc.). Back around the time that this research was first published I was engaged in research on Age of English Acquisition (AEA) effects on recall and recognition tests as well as examining the relationship of AEA to subject characteristics such as their SAT scores. While I found no relationship between AEA and recall/recognition, I did find a strong relationship between AEA and SAT scores (particularly verbal SAT). In an attempt to identify which component of the SAT AEA was related to, I actually used a "practice" SAT test as a filler activity between list learning and a memory test of the lists. During the construction of the SAT filler test (it consisted of antonyms, sentence completion, analogies, and reading comprehension questons) I recognized the text from one of the reading comprehension passages as being from the Charles Dickens' novel "Great Expectations". I admit that it had been a long time since I read the novel but I had seen the British film version of the novel several times and I found that I could answer most of the questions without reference to the provided text. It occurred to me that if the actual SAT used text from common sources like novels (my memory is that that was the practice at least circa 1990 -- one way to maintain "ecological validity" of the test items), then one could use their knowledge/familiarity with those sources -- even if it were only through movies of them -- to answer the reading comprehension questions. I believe the intent of the text developers is that a reader would develop a schema based on the reading of the test passage but, at least in this instance, didn't appreciate that such a schema could be already exist because of prior experience with the material. So, were the items/questions on the passage from "Great Expectations" "bad items"? I guess that depends upon how one defines what a "bad item" is. Clearly, people can answer reading comprehension questions in situations like this one with minimal reading of the test passage. In the context of the AEA research I engaged in, I realized that the correlation between AEA and reading comprehension may say less about intrinsic reading ability than it does about a person's general faimiliarity with the cultural artifacts of the English-speaking world. Hmmmm, maybe someone should do an experiment when (a) subjects are pre-screened as to whether they watch and closely follow a popular TV program (e.g., "Desperate Housewives") and (b) provide them a reading comprehension test where one of the passage provides a description from one of the episodes. It wouldn't be surprising if fans of the TV show performed better on the test relative to people who didn't follow the program but would this mean that the multiple choice iterms used in the test were "bad"? -Mike Palij [EMAIL PROTECTED] New York University --- You are currently subscribed to tips as: [email protected] To unsubscribe send a blank email to [EMAIL PROTECTED]
