Re: [Clips] Read two biometrics, get worse results - how it works
This problem has implications for "sensor fusion" (the latest hot topic) in IDS; for example when combining host logs (HIDS) with NIDS alerts. The risk of false positives is particularly relevant when you try to write signatures that match similar but unknown bad stuff, and false negatives when dealing with novel "zero day" attacks. Sometimes it's not always clear how to generalize to all the forms an attack could take (a problem compounded in a closed source environment), proper decoding of a vulnerable protocol could itself be dangerous or resource-prohibitive at wire speeds, so you end up with a compromise. Assuming that one wants to run tests at the equal error rate is a nice way to reduce the classification error relationship to a single statistic for analysis, but it's an assumption that may not hold in an operational environment. If the false negative costs a life, and a false positive means inconveniencing someone, you may want to run on the conservative side of the equal error rate. An interesting and somewhat related phenomenon is the "base rate fallacy", which involves a positive test for a rare condition. Assume 1 in ~1 people have a condition, and the test for it gives a false positive 1 in 100 times. Assume you test positive - intuition tends to tell us that we likely have the condition (after all, the test correct 99% of the time). In fact for every true positive, there are 10,000 opportunities for the false positive, so in fact your chances of actually having the condition are merely 1 in 100. For a prolonged explanation, see this paper: http://www.raid-symposium.org/raid99/PAPERS/Axelsson.pdf -- http://www.lightconsulting.com/~travis/ -><- "We already have enough fast, insecure systems." -- Schneier & Ferguson GPG fingerprint: 50A1 15C5 A9DE 23B9 ED98 C93E 38E9 204A 94C2 641B - The Cryptography Mailing List Unsubscribe by sending "unsubscribe cryptography" to [EMAIL PROTECTED]
Re: [Clips] Read two biometrics, get worse results - how it works
At 08:34 PM 10/19/2005, R.A. Hettinga wrote: A regular correspondent (thanks, you know who you are) points us to some calculations by John Daugman, originator of the Daugman algorithms for iris recognition. These ought to provide disturbing reading for Home Office Ministers who casually claim that by using multiple biometrics (http://www.theregister.co.uk/2005/10/17/mcnulty_fingers_id_problem/) you'll get a better result than by using just the one. Although that may seem logical, it turns out that it it isn't, necessarily. A man with one watch knows what time it is; a man with two watches is never quite sure. -- Attributed by Quote Garden to Lee Segall Steve - The Cryptography Mailing List Unsubscribe by sending "unsubscribe cryptography" to [EMAIL PROTECTED]
Re: [Clips] Read two biometrics, get worse results - how it works
On 10/19/05, R.A. Hettinga <[EMAIL PROTECTED]> wrote: > > [EDIT] > Daugman presents > (http://www.cl.cam.ac.uk/users/jgd1000/combine/combine.html) the two rival > intuitions, then does the maths. On the one hand, a combination of > different tests should improve performance, because more information is > better than less information. But on the other, the combination of a strong > test with a weak test to an extent averages the result, so the result > should be less reliable than if one were relying solely on the strong test. > I believe the Daugman results are correct only when one accepts results where the tests disagree. That is, if the first test returns positive and the second test returns negative, you chose the overall results to be positive or negative as opposed to "do over until they agree". Of course, in real life with knowledge of the physics of the tests and the ability to pull out non-boolean results, one may be able to remove many of the "do over" results to keep from annoying the test subjects. -Michael Heyman - The Cryptography Mailing List Unsubscribe by sending "unsubscribe cryptography" to [EMAIL PROTECTED]
Re: [Clips] Read two biometrics, get worse results - how it works
RAH, et al., It is true that one can combine two diagnostic tests to a worse effect than either alone, but it is not a foredrawn conclusion. To take a medical example, you screen first with a cheap test that has low/no false negatives then for the remaining positives you screen with a potentially more expensive test that has low/no false positives. There is a whole health policy & management literature on this. I reproduce the barest precis of same below, assuming the reader can manage to view it in a fixed width font while respecting my hard carriage returns as writ. --dan cheat sheet on terminology of medical diagnostic testing _ \ the true situation \ \+ - +---+---+--- | | | + | a | b | a+b what the | | | diagnostic +---+---+--- test returns | | | - | c | d | c+d | | | +---+---+--- | | | | a+c | b+d | t true positives a = positive testers who have disease true negatives d = negative testers who are without disease false positives b = positive testers who are without disease false negatives c = negative testers who have disease prevalence (a+c)/t = fraction of population that has disease sensitivity a/(a+c) = what fraction of those with disease test positive specificity d/(b+d) = what fraction of those without disease test negative predictive value positive a/(a+b) = what fraction of positive tests have disease predictive value negative a/(a+b) = what fraction of negative tests are without disease Notes: Information retrieval people know sensitivity as "recall" and predictive value positive as "precision." Screening with a cheap test with high sensitivity then an expensive test with high specificity is often the best (most cost effective) strategy. - The Cryptography Mailing List Unsubscribe by sending "unsubscribe cryptography" to [EMAIL PROTECTED]
[Clips] Read two biometrics, get worse results - how it works
--- begin forwarded text Delivered-To: [EMAIL PROTECTED] Date: Wed, 19 Oct 2005 23:32:55 -0400 To: Philodox Clips List <[EMAIL PROTECTED]> From: "R.A. Hettinga" <[EMAIL PROTECTED]> Subject: [Clips] Read two biometrics, get worse results - how it works Reply-To: [EMAIL PROTECTED] Sender: [EMAIL PROTECTED] <http://www.theregister.co.uk/2005/10/19/daugman_multi_biometrics/print.html> The Register Biting the hand that feeds IT The Register » Internet and Law » Digital Rights/Digital Wrongs » Original URL: http://www.theregister.co.uk/2005/10/19/daugman_multi_biometrics/ Read two biometrics, get worse results - how it works By John Lettice (john.lettice at theregister.co.uk) Published Wednesday 19th October 2005 14:47 GMT A regular correspondent (thanks, you know who you are) points us to some calculations by John Daugman, originator of the Daugman algorithms for iris recognition. These ought to provide disturbing reading for Home Office Ministers who casually claim that by using multiple biometrics (http://www.theregister.co.uk/2005/10/17/mcnulty_fingers_id_problem/) you'll get a better result than by using just the one. Although that may seem logical, it turns out that it it isn't, necessarily. Daugman presents (http://www.cl.cam.ac.uk/users/jgd1000/combine/combine.html) the two rival intuitions, then does the maths. On the one hand, a combination of different tests should improve performance, because more information is better than less information. But on the other, the combination of a strong test with a weak test to an extent averages the result, so the result should be less reliable than if one were relying solely on the strong test. (If Tony McNulty happens to be with us, we suggest he fetches the ice pack now.) "The key to resolving the apparent paradox," writes Daugman, "is that when two tests are combined, one of the resulting error rates (False Accept or False Reject rate) becomes better than that of the stronger of the two tests, while the other error rate becomes worse even than that of the weaker of the tests. If the two biometric tests differ significantly in their power, and each operates at its own cross-over point, then combining them gives significantly worse performance than relying solely on the stronger biometric. This is of particular relevance to the Home Office's current case for use of multiple biometrics, because its argument is based on the use of three types of biometric, fingerprint, facial and iris, which are substantially different in power. Daugman produces the calculations governing the use of two hypothetical biometrics, one with both false accept and false reject rates of one in 100, and the second with the two rates at one in 1,000. On its own, biometric one would produce 2,000 errors in 100,000 tests, while biometric two would produce 200. You can treat the use of two biometrics in one of two ways - the subject must be required to pass both (the 'AND' rule) or the subject need only pass one (the 'OR' rule). Daugman finds that under either rule there would be 1,100 errors, i.e. 5.5 times more errors than if the stronger test were used alone. He concludes that a stronger biometric is therefore better used alone than in combination, but only when both are operating at their crossover points. If the false accept rate (when using the 'OR' rule) or the false reject rate (when using the 'AND' rule) is brought down sufficiently (to "smaller than twice the crossover error rate of the stronger test", says Daugman) then use of two can improve results. If we recklessly attempt to put a non-mathemetical gloss on that, we could think of the subject having to pass two tests (in the case of the 'AND') rule of, say, facial and iris. Dropping the false reject rate of the facial test (i.e. letting more people through) in line with Daugman's calculations would produce a better result than using iris alone, but if the facial system rejects fewer people wrongly, then it will presumably be accepting more people wrongly. Which suggests to us that simply regarding a second or third biometric as a fall back to be used only if earlier tests fail constructs a scenario where the combined results will be worse than use of the single stronger test, because in such cases the primary biometric test would have to be sufficiently strong to stand on its own, because you won't always be using the second or third test. The deployment of biometric testing equipment in the field is also likely to have a confusing effect on relative error rates, because environmental factors will tend to impact the different tests to different degrees. Poor lighting may have an effect on iris and facial but not on fingerprint, while the aircon breaking down may produce greasy fingers and puffy red faces, but leave
