Re: memcached SolrCache implementation?

2007-01-14 Thread Chris Hostetter

: I'd like to see about making a SolrCache implementation that uses the
: memcached library @ http://www.whalin.com/memcached/. I believe that
: this would be useful for replicated sites, allowing all the search nodes
: to use a shared global cache.

it's not something i've ever considered, mainly because i've always
thought of the power of a SolrCache being in it's locality.

the one thing you'd really have to watch out for is the machines being in
sync ... i don't know if memcached has the notion of managing multiple
dynamicly named caches, but you need to make sure that when a machine asks
for a key, it's using the correct cache based on the IndexReader it
currently has open -- if another slave gets snapshots from the master a
little bit faster and starts populating the cache with DocSets using the
new docIds you don't want to be getting those on your slightly behind
slave.

: seems like this is doable. I don't believe it's possible to implement
: autowarming, as there doesn't seem to be a way to get a list of cached
: keys from the memcached library.

you could let memcached cache the key=>vals but you could maintain an
inmemory list of the recent (or frequently) accessed keys on this tier --
the keys are really all you need for autowarming.

: If so, I'm not certain how to pass up additional arguments to a custom
: SolrCache implementation - I'd like to replace queryResultCache with my
: implementation, but will need to provide additional configuration -
: memcache server string, etc. Is it permissible to add custom attributes
: to the queryResultCache configuration node?

off the top of my head, i'm pretty sure that you can add any attribute you
want and it will get passed to the init method in the "Map args" .. i'm
not 100% certain though.



-Hoss



memcached SolrCache implementation?

2007-01-12 Thread Shawn Gervais

Hi devs,

I'd like to see about making a SolrCache implementation that uses the 
memcached library @ http://www.whalin.com/memcached/. I believe that 
this would be useful for replicated sites, allowing all the search nodes 
to use a shared global cache.


From my reading of the LRUCache code and the SolrCache interface, it 
seems like this is doable. I don't believe it's possible to implement 
autowarming, as there doesn't seem to be a way to get a list of cached 
keys from the memcached library.


Can someone comment on whether this seems like a good idea? I would like 
to dig in the Solr internals, and this seems like a good starting point.


If so, I'm not certain how to pass up additional arguments to a custom 
SolrCache implementation - I'd like to replace queryResultCache with my 
implementation, but will need to provide additional configuration - 
memcache server string, etc. Is it permissible to add custom attributes 
to the queryResultCache configuration node?


Thanks for the feedback

-Shawn