Membase has memcached built in, if you are looking for a key-value database. Provides in-memory caching like memcached, but also persists data to durable media automatically. 100% on-the-wire compatible with memcached. Currently lacks the query capability of MongoDB (beyond the obvious primary key based "query") ... but not for long.
On Fri, Oct 1, 2010 at 7:38 AM, ligerdave <[email protected]> wrote: > again, you dont need to generate all at once. whenever a request asked > for certain objects, get it in db(if not in cache) and store in cache. > loading locally means you would have wasted space for duplicates and > you need to face synchronization issues. > > i think in your case, you can use a nosql(key-value db) solution > called mongodb. it has a memory-mapped file system and it supports > some easy queries to allow you doing some type sql-like operations. > > the goal is to have all information loaded into memory(cached) to > speed up your app. how to smartly load the objects is the key. > > > > > On Sep 30, 11:59 pm, parsa <[email protected]> wrote: > > > i suspect that you already have a DB. am i right? > > > > Yes that's where I'm getting the data from, it's on another server > > though. > > > > > How do you generate this key-value map? All at once, or can you compute > each > > > individual value given a key? > > > > I generate it all at once from the DB and it's an expensive process. > > > > > How do you use this map? All at once, or a few values for each web > service > > > request? > > > > Each request only needs parts of the map, not all of it. But as the > > number of simultaneous requests grows to somewhere near 500, there's a > > chance of using 90% of the map. > > > > > How does the map change? All at once, or do you know which specific > keys > > > need to be invalidated? > > > > It doesn't change in run-time. It changes on a schedule once in a > > month. > > > > I think caching is not the way to go for me. I've looked into key- > > value databases but the problem is the algorithm that's triggered with > > each request (think of some searching) requires a specific type of > > data which is a Trie or prefix tree. Currently, I generate the map > > once in a singleton object inside the servlet container and give > > references to it for each request and it works. But what I'm saying > > is, maybe it's better to hold the data as a normal key-value map, then > > when each request arrives, generate a Trie out of it and run the > > algorithm with that Trie. (some sort of lazy loading) > > > > Thanks for your tips, fellas. > > > > On Sep 30, 11:38 pm, Henrik Schröder <[email protected]> wrote: > > > > > > > > > How do you generate this key-value map? All at once, or can you compute > each > > > individual value given a key? > > > > > How do you use this map? All at once, or a few values for each web > service > > > request? > > > > > How does the map change? All at once, or do you know which specific > keys > > > need to be invalidated? > > > > > If you can re-compute single values easily, and if you only need a few > of > > > them per request, and if you will invalidate single keys, then > memcached is > > > a good fit for your project. Every time you need a value, you first > check > > > the cache. If it's in the cache, great, you got it. If not, compute the > > > value, and put it in the cache. If a value changes, just remove it from > > > memcached, or compute it and put in the new value immediately. > > > > > How often does the map change? If it changes extremely rarely, you > could > > > just cache the map in application memory on each individual webserver > > > instead, and have some mechanism for invalidating all of them at once. > > > > > Remember that memcached is a cache, it is not a permanent data store. > > > Putting an item into it in no way guarantees that you will get it out, > it > > > only guarantees that if you get something out, it will be the latest > version > > > of the item. > > > > > /Henrik > > > > > On Wed, Sep 29, 2010 at 18:18, parsa <[email protected]> wrote: > > > > Hey fellas, > > > > > > I have a large key-value map that I want to serve in a web service > > > > application. I want to keep a single instant of this map inside the > > > > memory (around 600mb footprint) and let every request that is made to > > > > the service use the very same object. I'm new to memcached and to be > > > > honest, caching in general. So is it better to keep the object in the > > > > memory as a whole or to add key-values to the cache separately? (btw > > > > I'm using Scala on Lift)
