(This is a little different -- the estimate isn't in [0,1], it is in [1,1]! The commentary is right, in the abstract.)
On Tue, Oct 25, 2011 at 9:26 PM, Ted Dunning <[email protected]> wrote: > Speaking statistically, AAD has some interesting issues when you are trying > to estimate a boolean value. > > In this framework, you produce an estimate in [0,1] of an actual value that > is in {0,1}. If you guess binary results, you will have an error of 0 or 1. > If you guess intermediate values, you are guaranteed to have a non-zero > error. If the actual probability of getting a 1 is p and you estimate x, > then the expected value of AAD is (1-p) x + p (1-x) = > x-px + p-px = p + x (1-2p). If p<0.5, this is minimized by setting x = 0 > and if p>0.5, by setting x = 1. Thus, you minimize AAD by only guessing > binary values (which our recommenders never do, btw). > > As such, if you really want to use AAD as a quality metric, you may want to > put in a step that clamps the output to 0 or 1 before evaluating.
