Yes, that's quite small. As a reference. I'm currently writing up a case study on a data set with 130K users and 160K items and recommendation time is from 10ms to 200ms, depending on the algorithm. Your use case seems to require 200-400ms per recommendation -- on a 1-core machine.
I'd recommender the -d64 flag if your system is 64-bit, but that's marginal. I think the big slowdown is probably the translation from strings to longs and back. What's your DataModel like? getting good performance in the case where you need to do translation can be tricky and I suspect this is the issue. On Tue, Nov 24, 2009 at 7:10 PM, Otis Gospodnetic <[email protected]> wrote: > Hello, > > I've been using Taste for a while, but it's not scaling well, and I suspect > I'm doing something wrong. > When I say "not scaling well", this is what I mean: > * I have 1 week's worth of data (user,item datapoints) > * I don't have item preferences, so I'm using the boolean model > * I have caching in front of Taste, so the rate of requests that Taste needs > to handle is only 150-300 reqs/minute/server > * The server is an 8-core 2.5GHz 32-bit machine with 32 GB of RAM > * I use 2GB heap (-server -Xms2000M -Xmx2000M -XX:+AggressiveHeap > -XX:MaxPermSize=128M -XX:+CMSClassUnloadingEnabled > -XX:+CMSPermGenSweepingEnabled) and Java 1.5 (upgrade scheduled for Spring) > > ** The bottom line is that with all of the above, I have to filter out less > popular items and less active users in order to be able to return > recommendations in a reasonable amount of time (e.g. 100-200 ms at the > 150-300 reqs/min rate). In the end, after this filtering, I end up with, > say, 30K users and 50K items, and that's what I use to build the DataModel. > If I remove filtering and let more data in, the performance goes down the > drain. > > My feeling is 30K users and 50K items makes for an awfully small data set and > that Taste, esp. at only > 150-300 reqs/min on an 8-core server should be much faster. I have a feeling > I'm doing something wrong and that Taste is really capable of handling more > data, faster. Here is the code I use to construct the recommender: > > idMigrator = LocalMemoryIDMigrator.getInstance(); > model = MyDataModel.getInstance("itemType"); > > // ItemSimilarity similarity = new LogLikelihoodSimilarity(model); > similarity = new TanimotoCoefficientSimilarity(model); > similarity = new CachingUserSimilarity(similarity, model); > > // hood size is 50, minSimilarity is 0.1, samplingRate is 1.0 > hood = new NearestNUserNeighborhood(hoodSize, minSimilarity,similarity, > model, samplingRate); > > recommender = new GenericUserBasedRecommender(model, hood, similarity); > recommender = new CachingRecommender(recommender); > > What do you think of the above numbers? > > Thanks, > Otis >
