Yeah, I have really struggled with this piece. The original vision was to support real-time updates -- recommendations always use all the latest data. It was quickly clear that was far too slow and wasteful. So I bolted on caching... and started putting it everywhere. And it quickly became a bit of a mess to manage correctly expiring cache entries and so on.
I ended up, over time, with the reasonable "Refreshable" scheme which lets components update their state (and caches) after having efficiently had their downstream dependencies update. You've found a part where that kind of breaks down. I suppose the idea is that updates are driven solely from upstream components. The Recommender decides when to update and passes that message down. Under that view, the behavior is correct -- the FileDataModel is welcome to update itself, and eventually the upstream components will care to re-read from it. But then perhaps this automatic reloading is pointless. It should happen on demand, perhaps with a limit to avoid re-reading updates too frequently or something. Let me pause and ask for your thoughts? On Wed, Jan 28, 2009 at 6:51 PM, Otis Gospodnetic <[email protected]> wrote: > Hi, > > I'm trying out CachingRecommender to see how much it improves performance for > me (it does a lot when the cache is all warmed up). > One thing that I assumed would happen automatically is that the > CachingRecommender will be refreshed when the underlying file (FileDataModel) > is refreshed. Instead, I do see the FDM re-reading the input file when it > detects the change, but my recommendations remain cached forever, as I'm not > hitting memory limits. > > I've been looking at the code, but can't see a clean way to "listen" to the > FDM and find out when it finished re-reading the file, so I can clear my > CachingRecommender. Is there an existing way this should be done that I'm > just not seeing in the code? > > Ah, I see another thing now. Consider this: > > Recommender recommender = new > BooleanUserGenericUserBasedRecommender(model, hood, similarity); > recommender = new CachingRecommender(recommender); > > Now if I did somehow find a way to clear the caching recommender, I imagine > I'd want to call one of its clear(...) methods, and this requires a cast back > to CachingRecommender. Is there a better way? > > Thanks, > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > >
