Thanks, I admit this is pushing the limits of my knowledge of stats. Maybe you can help me offline here. What's the LLR function, and how does this table of four values related to k1 and k2 in the formula in section 8? is that formula OK to implement, since I think I do get that one.
The chi-squared stat is a positive value right? then yes to map into [-1,1] to make it correlation-like I'd need a simple transformation or two like what you describe. On Wed, May 21, 2008 at 2:49 PM, Ted Dunning <[EMAIL PROTECTED]> wrote: > Yes. If you have things A and B and counts k_ab for when A and B occur > together, k_a for when A appears with or without B, k_b similar to k_a but > for B and N which is the total observations then you build the table: > > k_ab k_a - k_ab > k_b - k_ab N - k_a - k_b + k_ab > > and then you can apply the LLR function directly. > > It is often helpful to report sqrt(-2 log lambda) with a sign according to > whether k_ab is greater of less than expected, i.e. > > signum( k_ab / k_b - k_a / N ) * sqrt(- 2 log lambda) > > Also, for computation of - 2 log lambda, it is usually easier to compute it > in terms of mutual information which is in turn expressed in terms of > entropy: > > - 2 log lambda = N * MI = N * ( H(K) - H(rowSums(K)) - H(colSums(K)) ) > > H(X) = - sum (X / sum(X)) logSafe (X / sum(X)) > > logSafe(x) = log(x + (x == 0)) > > The resulting score is not directly suitable as a distance measure, but is > very handy for masking co-occurrence matrices. > > On Wed, May 21, 2008 at 11:22 AM, Sean Owen <[EMAIL PROTECTED]> wrote: > >> Got it, so do I have it right that you suggest defining the >> "correlation" as really the chi-squared statistic, the -2 log lamba >> formula? k1 is the number of items 'preferred' by user 1, and n1 is >> the total number of items in the universe, and likewise for k2/n2? so >> n1 == n2? >> >> Simple is cool with me, to start. This sounds more sophisticated than >> a simple intersection/union approach. I could add that algorithm too >> for kicks. >> >> On Wed, May 21, 2008 at 12:58 PM, Ted Dunning <[EMAIL PROTECTED]> >> wrote: >> > Correlation (per se) between such sparse binary vectors can be very >> > problematic. >> > >> > This is a general problem with this kind of data and really needs to be >> > handled directly. Not clicking on an item is much less informative than >> > clicking on an item (so little time, so much to click). Any system you >> > build has to deal with that and with coincidence. For instance, raw >> > correlation gives 100% match for two people who happen to have clicked on >> > the same single item. IF that item is a very popular one, however, this >> is >> > not a very interesting fact. >> > >> > One very simple way of dealing with this was described in >> > http://citeseer.ist.psu.edu/29096.html . Since then, I have found >> other, >> > more comprehensive techniques but they are considerably more complex. >> > > > > -- > ted >
