I just did some detailed testing of OnlineAuc. Somewhat surprisingly (to me, at least) the FAIR policy was decidedly sub-optimal and FIFO is the best.
Also, at 10,000 samples and a window of 10, OnlineAuc is almost identically accurate as Auc. A window of 100 increases the accuracy very minutely, but not anything like what it costs. Decreasing the window to 2 decreases accuracy noticeably for the FAIR and RANDOM policies, but causes almost no change for the FIFO policy. As a result of these tests, I have set the window to 10 and the default policy to FIFO.
