I have not read the article; I will if I have some time in the near future,
(LOL) but you are making the assumption that the passages in general
might have
familiar information in them. Actually, we don't know that that is the
case. So
I will go back to my original suggestion that this article may provide
evidence
that one need not have read a passage to answer the questions based on the
passage, if there is knowledge in a person's general knowledge base to
provide
asnwers.
On the other hand, upon further reflection of the abstract contents, I
think you
are correct in your assessment of this article, in that the items
themselves are
probably quite well thought out and well written. It is only the
context of this
test and this research study that the items come into question.
Annette
Quoting Mike Palij <[EMAIL PROTECTED]>:
On Wed, 01 Mar 2006 09:50:40 -0800, Annette Taylor, Ph. D. wrote:
I found the reference I was thinking of:
Katz, S., Lautenschlager, G. J., Blackburn, A. B. & Harris, F. H.
(1990). Answering reading comprehension items without passages
on the SAT. Psychological Science, 1, 122-127.
Here is the abstract:
Examined whether the Reading Comprehension (RC) task on the
Scholastic Aptitude Test (SAT) measures factors that are unrelated
to RC. Two experiments were conducted with 197 college students.
Performance of Ss on the RC task was well above chance when
reading passages were deleted. Moreover, Ss and test items
performed similarly with or without the passages: Individual performance
correlated with verbal SAT score, and the difficulty of items belonging
to a passage correlated with a normative measure based on equated delta.
Thus, performance on the RC task appears to depend substantially on
factors having nothing to do with understanding the passages normally
accompanying test questions. (PsycINFO Database Record (c) 2005
APA, all rights reserved)
I don't think that this research really is relevant to the issue of
bad multiple choice items/tests (which I interpret as reflecting
questions and answers that are vague, ambiguous, are consistent
with multiple interpretations, etc.). Back around the time that
this research was first published I was engaged in research on
Age of English Acquisition (AEA) effects on recall and recognition
tests as well as examining the relationship of AEA to subject
characteristics such as their SAT scores. While I found no
relationship between AEA and recall/recognition, I did find a
strong relationship between AEA and SAT scores (particularly
verbal SAT). In an attempt to identify which component of the
SAT AEA was related to, I actually used a "practice" SAT test
as a filler activity between list learning and a memory test of the
lists.
During the construction of the SAT filler test (it consisted of
antonyms, sentence completion, analogies, and reading
comprehension questons) I recognized the text from one
of the reading comprehension passages as being from the
Charles Dickens' novel "Great Expectations". I admit that
it had been a long time since I read the novel but I had seen
the British film version of the novel several times and I found
that I could answer most of the questions without reference
to the provided text. It occurred to me that if the actual
SAT used text from common sources like novels (my memory
is that that was the practice at least circa 1990 -- one way
to maintain "ecological validity" of the test items), then one
could use their knowledge/familiarity with those sources --
even if it were only through movies of them -- to answer
the reading comprehension questions. I believe the intent
of the text developers is that a reader would develop a
schema based on the reading of the test passage but, at
least in this instance, didn't appreciate that such a schema
could be already exist because of prior experience with
the material.
So, were the items/questions on the passage from "Great
Expectations" "bad items"? I guess that depends upon how
one defines what a "bad item" is. Clearly, people can
answer reading comprehension questions in situations like
this one with minimal reading of the test passage. In the
context of the AEA research I engaged in, I realized that
the correlation between AEA and reading comprehension
may say less about intrinsic reading ability than it does about
a person's general faimiliarity with the cultural artifacts of
the English-speaking world.
Hmmmm, maybe someone should do an experiment when
(a) subjects are pre-screened as to whether they watch
and closely follow a popular TV program (e.g., "Desperate
Housewives") and (b) provide them a reading comprehension
test where one of the passage provides a description from
one of the episodes. It wouldn't be surprising if fans of the
TV show performed better on the test relative to people
who didn't follow the program but would this mean that the
multiple choice iterms used in the test were "bad"?
-Mike Palij
[EMAIL PROTECTED]
New York University
---
You are currently subscribed to tips as: [EMAIL PROTECTED]
To unsubscribe send a blank email to [EMAIL PROTECTED]
Annette Kujawski Taylor, Ph. D.
Department of Psychology
University of San Diego
5998 Alcala Park
San Diego, CA 92110
[EMAIL PROTECTED]
---
You are currently subscribed to tips as: [email protected]
To unsubscribe send a blank email to [EMAIL PROTECTED]
