Yes. If you have things A and B and counts k_ab for when A and B occur
together, k_a for when A appears with or without B, k_b similar to k_a but
for B and N which is the total observations then you build the table:
k_ab k_a - k_ab
k_b - k_ab N - k_a - k_b + k_ab
and then you can apply the LLR function directly.
It is often helpful to report sqrt(-2 log lambda) with a sign according to
whether k_ab is greater of less than expected, i.e.
signum( k_ab / k_b - k_a / N ) * sqrt(- 2 log lambda)
Also, for computation of - 2 log lambda, it is usually easier to compute it
in terms of mutual information which is in turn expressed in terms of
entropy:
- 2 log lambda = N * MI = N * ( H(K) - H(rowSums(K)) - H(colSums(K)) )
H(X) = - sum (X / sum(X)) logSafe (X / sum(X))
logSafe(x) = log(x + (x == 0))
The resulting score is not directly suitable as a distance measure, but is
very handy for masking co-occurrence matrices.
On Wed, May 21, 2008 at 11:22 AM, Sean Owen <[EMAIL PROTECTED]> wrote:
> Got it, so do I have it right that you suggest defining the
> "correlation" as really the chi-squared statistic, the -2 log lamba
> formula? k1 is the number of items 'preferred' by user 1, and n1 is
> the total number of items in the universe, and likewise for k2/n2? so
> n1 == n2?
>
> Simple is cool with me, to start. This sounds more sophisticated than
> a simple intersection/union approach. I could add that algorithm too
> for kicks.
>
> On Wed, May 21, 2008 at 12:58 PM, Ted Dunning <[EMAIL PROTECTED]>
> wrote:
> > Correlation (per se) between such sparse binary vectors can be very
> > problematic.
> >
> > This is a general problem with this kind of data and really needs to be
> > handled directly. Not clicking on an item is much less informative than
> > clicking on an item (so little time, so much to click). Any system you
> > build has to deal with that and with coincidence. For instance, raw
> > correlation gives 100% match for two people who happen to have clicked on
> > the same single item. IF that item is a very popular one, however, this
> is
> > not a very interesting fact.
> >
> > One very simple way of dealing with this was described in
> > http://citeseer.ist.psu.edu/29096.html . Since then, I have found
> other,
> > more comprehensive techniques but they are considerably more complex.
>
--
ted
