This is a great breakdown, thank you David! A few comments: On Mon, Jun 16, 2014 at 9:59 AM, David Leon Gil <[email protected]> wrote: > > ## Summary > > The overall goal: Determine whether fingerprint format affects the > reliability of user comparison of fingerprints. >
I think our real-world goal is "helps users ensure they're communicating with the intended party" We can specify that we're only looking at solutions here that involve fingerprint comparison but it's worth keeping the real-world goal in mind. *Factor B.* Incentive to reject fakes: > > 1. None > 2. Desire to "do well" or please experimenter > 3. Game-like incentive (e.g., Mechanical Turk performance-based > compensation) > 4. 'Real-world' privacy-preservation-like incentive (e.g., belief that > security of answers to personally sensitive questions rests on correct > performance) > Not sure if #3 encapsulates this, but you can use monetary incentives in an experiment. > ### Factors that are measurable, but hard to select for > > *Factor E.* Subject type: > > 1. Pure novice subjects (e.g., an Internet user who doesn't know what a > fingerprint is, doesn't understand the cost of generating collisions, and > has never attempted this tasks) > 2. Educated novice subjects > 3. Experienced subjects > 4. Educated and experienced subjects > This can be approximated by giving users more instructions in one treatment than another, so perhaps this belongs above. *Factor F.* Learning style: > > (Needs research; likely needs to be measured and results normalized to > population prevalences. Note that I believe that there is substantial > evidence that a one-size-fits-all fingerprint verification format will be > inferior to allowing users to choose a preferred fingerprint format. Here, > it might be interesting to do an experiment with feedback; e.g., have a > subject choose a fingerprint format to verify, provide feedback on > accuracy, then allow choosing another format, etc.) > I would discourage any attempt to vary this factor. For the conceivable future a one-size-fits-all format is necessary. Missing factor H? The similarly of the fake fingerprints to the expected genuine ones. ## The proposed experiment > > As I understand it, the consensus is that an experiment that is likely to > have discriminatory power among fingerprint types is infeasible to conduct > in a realistic setting. (I.e., the 'head fake' type scenarios.) I'd tend to > agree. > > So, the proposed experiment is, approximately: A1/B3/C1. > > For that experiment, I'd note that the actual probability of a fake > fingerprint (and perhaps the 'goodness' of the fake) has to vary so to > allow extrapolation to the zero-cheater case. (Though I'd expect that very > few participants will cheat unless the compensation scheme is extremely > imbalanced.) > Why are participants cheating? I think all errors should be introduced by experiments rather than have subjects insert their own error.s ## The gold-standard experiment > > A large trial among users of messaging software that requires fingerprint > verification, in which errors are introduced (with some small probability) > in fingerprints. > Personally I think this is the way to go.
_______________________________________________ Messaging mailing list [email protected] https://moderncrypto.org/mailman/listinfo/messaging
