- I want to comment a little more thoroughly about the lines I cited:
what Garson said about inference, and his citation of Olkey.
On Thu, 22 Feb 2001 18:21:41 -0500, Rich Ulrich <[EMAIL PROTECTED]>
wrote:
[ snip, previous discussion ]
me >
> I think that Garson is wrong, and the last 40 years of epidemiological
> research have proven the worth of statistics provided on non-random,
> "observational" samples. When handled with care.
> ================
> From G. David Garson, "PA 765 Notes: An Online Textbook."
>
> On Sampling
> http://www2.chass.ncsu.edu/garson/pa765/sampling.htm
>
> Significance testing is only appropriate for random samples.
>
> Random sampling is assumed for inferential statistics
> (significance testing). "Inferential" refers to the fact
> that conclusions are drawn about relationships in the data
> based on inference from knowledge of the sampling
> distribution. Significance tests are based on a sampling
> theory which requires that every case have a chance of being
> selected known in advance of sample selection, usually an
> equal chance. Statistical inference assesses the
> significance of estimates made using random samples. For
> enumerations and censuses, such inference is not needed
> since estimates are exact. Sampling error is irrelevant and
> therefore inferential statistics dealing with sampling error
> are irrelevant.
- I agree with most of what he says, throughout; there will be a
matter of nuances on interpretation and actions.
For enumerations and censuses, a limited sort of statistics on 'finite
populations,' he says sampling error is irrelevant. Irrelevant is a
good and fitting word here. This is not 'illegal and banned,' but
rather 'unwanted and totally beside the point.'
Garson >
> Significance tests are sometimes applied
> arbitrarily to non-random samples but there is no existing
> method of assessing the validity of such estimates, though
> analysis of non-response may shed some light. The following
> is typical of a disclaimer footnote in research based on a
> non random sample:
Here is my perspective on testing, which does not match his.
- For a randomized experimental design, a small p-level on
a "test of hypothesis" establishes that *something* seemed
to happen, owing to the treatment; the test might stand
pretty-much by itself.
- For a non-random sample, a similar test establishes that
*something* seems to exist, owing to the factor in question
*or* to any of a dozen factors that someone might imagine.
The test establishes, perhaps, the _prima facie_ case but the
investigator has the responsibility of trying to dispute it.
That is, it is an investigator's responsibility (and not just an
option) to consider potential confounders and covariates.
If the small p-level stands up robustly, that is good for the
theory -- but not definitive. If there are vital aspects or factors
that cannot be tested, then opponents can stay unsatisfied,
no matter WHAT the available tests may say.
Garson >
> "Because some authors (ex., Oakes, 1986) note the use of
> inferential statistics is warranted for nonprobability
> samples if the sample seems to represent the population, and
> in deference to the widespread social science practice of
> reporting significance levels for nonprobability samples as
> a convenient if arbitrary assessment criterion, significance
> levels have been reported in the tables included in this
> article." See Michael Oakes (1986). Statistical inference: A
> commentary for social and behavioral sciences. NY: Wiley.
> ================
Garson is telling his readers and would-be statisticians a way to
present p-levels, even when the sampling doesn't justify it.
And, I would say, when the analysis doesn't justify it.
I am not happy with the lines -- The disclaimer does not assume
that a *good* analysis has been done, nor does it point to what
makes up a good analysis.
'... if the sample seems to represent the population'
seems to be a weak reminder of the proper effort to overcome
'confounding factors'; it is not an assurance that the effects
have proven to be robust.
So, the disclaimer should recognize that the non random sample
is potentially open to various interpretations; the present analysis
has attempted to control for several possibilities; certain effects
do seem robust statistically, in addition to being supported by
outside chains of inference, and data collected independently.
I suggested earlier that this is the status of epidemiological,
observational studies. For the most part, those studies have
been quite fruitful. But not always. They have been especially
likely to mislead, I think, when the designs pretend that binomial
variability is the only source of error in a large survey, and attempt
to interpret small effects.
--
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
