I don't think that this is so much a problem of ratings (which I dislike in practice as everybody must know by now).
Instead I see it as parallel problems. The first is a problem with small counts and adventitious results based on coincidental pairings. The second is the treatment of ratings as a continuous scale rather than as an ordered. The idea that you can subtract ratings and get a sensible number is not actually correct. In particular, when displaying recommended items, you typically only display items with high estimated values or values that have value for learning. This means that items with very low estimated values never get displayed at all. If you only get to display items with estimated ratings of 4.5 and above, then the consequences of an error of positive or negative 1/2 are very large for something with an estimate of 4.5, but an error of -2 for an item with a rating 3 is completely inconsequential. The problem of small counts probably dominates, but it is good to not lose sight of the other problem. Dealing with low counts with ratings requires a bit of thought and generally involves the complete rejection of techniques that are based on the assumption of normal distribution. Hence, correlation, normal chi^2 tests, t-tests, LSI, squared errors and so on are all out the window. Welcome instead latent variable models based on multinomials, Bayesian techniques and generalized log-likelihood ratios. On Thu, Aug 20, 2009 at 8:20 AM, Sean Owen <[email protected]> wrote: > > Well, if that's the only Lincoln book in the preferences that has any > defined similarity to the cookbook, the above algorithm says we should > return the weighted average of those preferences, weighted by > similarity. That's: (0.1 * 5.0) / 0.1. Or 5.0. The process says the > estimated rating is 5, and indeed there is some justification for > that. But it flies in the face of intuition. > > > Interesting. Takeaways? > > 1) The stock item-based recommender just has this weakness. I am open > to suggestion about alternate modes that could work around this. For > example: reject as a recommendation any item like this, where the > estimate would be based on a single data point. > > 2) Once again it seems that using preference values can actually get > one into trouble. Interesting. -- Ted Dunning, CTO DeepDyve
