This is a ubiquitous problem with coocurrence algorithms since they scale in the square of the number of occurrences most popular item.
The good news is that you learn everything there is to learn about that item if you look at just a sampling of the occurrences so sampling is your friend. If there is temporal structure, I tend to bias the sample toward recent items. Regarding the size, I have generally had an arbitrary cutoff attached to a configuration knob in my production systems. It is probably reasonable to set this limit to something like max(100, 20*log(max(N_users, N_items))). This isn't really any less arbitrary, but it will probably never need tweaking in normal use. On Wed, Jul 21, 2010 at 12:32 PM, Sean Owen <[email protected]> wrote: > Ah so it really is a function of those particular items. > Well we can probably modify this function to be smarter and cap, > somehow, the number of items considered. > I'm just struggling to figure out how to do so without drawing > arbitrary boundaries, like taking the top 100, etc. > > On Wed, Jul 21, 2010 at 8:46 PM, Sebastian Schelter > <[email protected]> wrote: > > I did some inspection on the grouplens dataset and it turned out that > > Ted was absolutely right. >
