yes I will be writing up my notes as I mentioned. directly on the github. These are from a professional UX research on experimental design for an academic study.
c On Thu, May 1, 2014 at 4:02 PM, Joseph Bonneau <[email protected]> wrote: >> 2 persons should be enough for a pilot to work out kinks, and 30 >> people would work in an academic setting for the statistics we want. > > > (a) Could you present a sketch of the experimental design? Exactly what > would each participant be asked to do step-by-step? > (b) Given that could you calculate a statistical power argument for why N=30 > is satisfactory given the effect size you're hypothesizing exists? > > Putting external validity concerns aside I'm very skeptical that N=30 is > close to enough. > >> A Mech Turk study could be complementary but it doesn't really >> replace. The idea for a scientific study would be to vary one variable >> at a time (the fingerprint mechanism) and a mechanical turk study >> varies potentially thousands (lighting, time of day, tiredness, >> screen, instructions, etc). > > > I disagree strongly. Several of the factors you mentioned (tiredness, time > of day) are going to vary regardless of an in-person vs. online study. These > factors are addressed by randomly assigning participants to treatments and > doing statistical testing to control for variance. Larger sample size is the > only real cure. Since they will vary in real life (people don't only check > fingerprints at 11 AM on a 17-inch screen), you're losing external validity > by fixing them, even if it may slightly reduce experimental noise. > > The instructions given are strictly easier to control in an online study and > you largely eliminate the potential for observer-expectancy effects. > > Most importantly with an online study you'll get a much more diverse > participant pool, and they'll do your task in the middle of tons of other > computer work in their own home, which is far closer to reality than driving > into a lab and sitting down to do research. > >> It could be good to do the scientific >> study to inform a larger mechanical turk study varying more variables. >> >> As for so many trials, the "learning effect" is a thing, and you can >> account for it statistically by varying the order the trials are >> presented in per subject. > > > Yes, but with N=30 you don't have nearly enough people to present every > possible permutation of 10 treatments, so you will be introducing potential > error. Also, this approach further weakens validity: In real life people > never "learn" how to compare fingerprints by trying 10 different ways to do > it in a row. _______________________________________________ Messaging mailing list [email protected] https://moderncrypto.org/mailman/listinfo/messaging
