Hi all, I did a little refactoring today to be able to inject customized ways of fetching the candidate items. I wrote another implementation that just returns all items not yet rated by the user. This won't be suitable for large datasets but it did quite well for the grouplens dataset (some testing results attached). I'm gonna create a patch so you can have a look at the refactoring and if you decide to commit it, it could be a suitable starting point for implementing Ted's proposed way of candidate item fetching.
Another advantage of that patch is that users could supply use-case specific implementations of candidate item fetching without having to subclass the recommender of their choice. --sebastian Tests for random users with different candidate item fetching strategies (grouplens dataset) User 1063 found 3605 items in 2376ms (current approach) found 3606 items in 1ms (all unknown items) User 3596 found 3575 items in 1889ms (current approach) found 3578 items in 2ms (all unknown items) User 3300 found 3343 items in 6603ms (current approach) found 3344 items in 0ms (all unknown items) User 924 found 3507 items in 4173ms (current approach) found 3507 items in 4ms (all unknown items) User 4505 found 3427 items in 4774ms (current approach) found 3427 items in 1ms (all unknown items) User 3378 found 3471 items in 4225ms (current approach) found 3471 items in 0ms (all unknown items) User 246 found 3673 items in 730ms (current approach) found 3677 items in 0ms (all unknown items) Am 22.07.2010 02:00, schrieb Ted Dunning: > This is a ubiquitous problem with coocurrence algorithms since they scale in > the square of the number of occurrences most popular item. > > The good news is that you learn everything there is to learn about that item > if you look at just a sampling of the occurrences so sampling is your > friend. If there is temporal structure, I tend to bias the sample toward > recent items. > > Regarding the size, I have generally had an arbitrary cutoff attached to a > configuration knob in my production systems. It is probably reasonable to > set this limit to something like max(100, 20*log(max(N_users, N_items))). > This isn't really any less arbitrary, but it will probably never need > tweaking in normal use. >
