Roger,
Statistical design of experiments is the specialty area devoted to
such endeavors. In such studies, the goal is often to study the result
of "crossing" both designable factors ("treatments" such as using M.
and not using M.) and controllable nuisance factors (such as the
sequence or time of day in which the treatments are done. Design often
enables you to reduce dramatically the sample sizes required by
combining the sampling error in the various "treatments".
http://en.wikipedia.org/wiki/Experimental_design
To answer your question directly, the choice between small sample
versus large sample analysis, when just 2 treatments are employed, is
almost entirely dependent on the two sample sizes, when independent
samples are used.
In most statistical studies, gathering a random sample is so costly
because an individual person ("subject") is a single sample, and each
person is so different that the sample error which leads "margin of
error" is huge. But in a computer study the single samples are single
runs of the one data set, typically, and random data sets can be
generated cheaply. A very widely used rule of thumb in all of
statistics is that 30 samples or more are considered "large", for any
randomly selected samples.
A really powerful experimental design in statistics is called a paired
difference design, wherein instead of computing the means of each of
two treatments, and then doing your large or small sample test on the
statistics based on the two means and standard deviations of those two
data sets, you calculate the difference between each individual pair
of identical data samples, and then statistically study whehter the
average difference is significantly different from 0. An exemplary
paired difference study would be to test to brands of tires by putting
both brands on the same car and computing the difference in tire wear
for each car. The sampling error is hugely reduced. And this is the
identical to most computer studies that force the exact same samples
on each of two "treatments".
--
(B=)
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
