I am a strong proponent (not surprisingly) of using log likelihood ratio (LLR) tests as a primary filter for finding good matches in cooccurrence analyses like this. It is very simple and has proven effective for many years in work I have done as well in other peoples work. Correlation based measures are very bad at extreme counts.
See here for a bit more detail: http://citeseer.ist.psu.edu/29096.html . I should have a blog entry up which illustrates how simple LLR tests are to implement (literally just a few lines of R and only twice or four times that many in Java). LLR can also be used very effectively for document routing using what are essentially variants on Naive Bayes classifiers where features are selected using LLR and weighted using corpus frequencies (usually something like log IDF). Other classifiers where you have vast numbers of possible features should work well based on the same work. On Fri, May 30, 2008 at 4:07 PM, Sean Owen <[EMAIL PROTECTED]> wrote: > ... > > Coming back to your message Ankur, yes I think the code already > supports what you suggest, in part. His "item similarity table" is the > "ItemCorrelation" you feed into GenericItemBasedRecommender. He > rightly suggests this should be precomputed, which is where > GenericItemCorrelation comes in -- it can just compute item-item > correlations based on any algorithm you want like PearsonCorrelation > and save them off. You could modify this to save only "strong" > correlations as you say. I can surely dig in here with you and write > code to do this if it's not clear. > > -- ted
