Paige Miller
>How exactly do you generalize the results from a "sample" of people who
>elect to respond to an opinion question, for example at http://cnn.com,
>to a larger population?
Johannes Hartig responded:
> If collecting data via internet is "not science" because of lacking
> "representativity" then most current research in psychology, where
> most samples are convenience samples of students, is "not science".
Kish (among other authors) points out that commonly the objective of a
survey is to be representative of a larger population, so that we can infer
from the sample to the population assuming the random mechanism by which
the sample was obtained.
The objectives of an experiment are quite different. The focus is on
the effect of a controlled intervention (treatment), and representativeness
of the subjects with respect to some broader population may be much less
important. In a given area of research, we sometimes may be reasonably
confident that the mechanisms operating in our convenient experimental
group are similar to those operating in the larger population, even
though the groups are different: for example, if hitting students on the
head or giving them a tranquilizer makes them remember less well, the
same may be true for other people. (For that matter, much basic science
research intended to lead to medical advances is conducted first on mice
or sheep, who are even less representative of the general human
population than are college students!) In other contexts, estimating the
average effect in a population may be more critical; for example, if a
new drug is being tested for broad release, tests must be performed in a
representative population of patients.
Some surveys actually have features that are more like experiments, in
this sense. I used to be fairly dismissive of typical market research
(telephone interviews with high noncontact and refusal rates, very little
follow-up, hastily conducted, etc.), compared to the high-quality government
surveys that I was used to using. This attitude was corrected by a colleague
guest-lecturing in my survey sampling class, who pointed out that these market
research surveys do a very good job at what they are meant for, which is to
get a sense of how some product or marketing innovation might play relative
to what is now out there. (and to do so quickly and cheaply!) No sensible
marketer would think that even a very well run survey could directly
determine how many units they would sell; even with the most representative
sample, a telephone response is not the same as putting down dollars at
the cash register. Nonetheless, survey results for a new product, in the
context of a series of surveys run in the same way for similar products
(market researchers have thousands of these on file), may be quite useful
in evaluating or targeting the product. Because the context is so
important, there is something of a comparative or experimental flavor to
the analysis (although not formal as in scientific experimentation).
Internet surveys may have something of the same usefulness. It will probably
be a long time (if ever) before control and response rates in these surveys
approach the point where we can consider them representative of the population
of Internet users (much less of the broader population). However, taken
in context of other, similar, surveys, they may have some usefulness in
measuring the level of interest in and support for an idea, product, or
whatever. I suspect that progress in the use of these surveys will depend
on developing some standardized methods for conducting them (which some
marketers are beginning to do, I believe) and then comparing results over
time. This does not leave a great role for the one-shot amateur survey
because even the best efforts are not going to make it representative of
a known population.
Alan Zaslavsky
Harvard Medical School
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
