Got it, so do I have it right that you suggest defining the "correlation" as really the chi-squared statistic, the -2 log lamba formula? k1 is the number of items 'preferred' by user 1, and n1 is the total number of items in the universe, and likewise for k2/n2? so n1 == n2?
Simple is cool with me, to start. This sounds more sophisticated than a simple intersection/union approach. I could add that algorithm too for kicks. On Wed, May 21, 2008 at 12:58 PM, Ted Dunning <[EMAIL PROTECTED]> wrote: > Correlation (per se) between such sparse binary vectors can be very > problematic. > > This is a general problem with this kind of data and really needs to be > handled directly. Not clicking on an item is much less informative than > clicking on an item (so little time, so much to click). Any system you > build has to deal with that and with coincidence. For instance, raw > correlation gives 100% match for two people who happen to have clicked on > the same single item. IF that item is a very popular one, however, this is > not a very interesting fact. > > One very simple way of dealing with this was described in > http://citeseer.ist.psu.edu/29096.html . Since then, I have found other, > more comprehensive techniques but they are considerably more complex.
