Hi Henrik, Eviction is not an issue, I don't plan to run out of memory (relatively small workset); as for server crash, well like any reasonable memcached user I planned on using consistent hashing to drive the choice of both a "master" and a few "replicas" with a quorum strategy for reads and writes.
Anyway, after thinking about it some more, it appears that memcached might be a premature choice here: a simple NoSQL database like Redis should provide way more throughput that I would ever need anyway, especially since moving to a more elastic setup means I can now scale horizontally (and thus sacrifice performance a bit to get reliability). Thanks for helping me out clarify my ideas. On Sunday, June 17, 2012 10:21:33 PM UTC+2, Henrik Schröder wrote: > > If you don't want to lose any data, how will you handle the case when > memcached evicts some of your data? How will you handle the case when a > memcached server crashes? Those two events are more likely to happen than > your planned cluster changes. > > > /Henrik > > On Sun, Jun 17, 2012 at 4:08 PM, Matthieu M. > <[email protected]>wrote: > >> I realize that I may be slightly pushing the limits. >> >> I have been looking at membase/couchdb/couchbase however the persistence >> options are so-so: I don't want to lose *any* data so the idea of "it will >> be written to the disk later" is not really satisfying. >> >> Unfortunately, like I said, I am aiming at highly volatile data, so I >> expect a read/write ratio of about 3/1; most components I have seen seem >> built in the expectation of a much more highly skewed ratio (say 100/1) and >> favor low-latency over persistence which does not suit me. >> >> Anyway, I have a couple leads already [1], I am just aiming at improving >> performance. >> >> >> [1] I have a couple options open to me, although only one does not >> require fiddling with memcached code: >> >> - Option 1 consists in versioning the list of memcached instances in the >> cluster and introduce a "pause" in the system when a change to the list is >> done so that all clients have the time to realize it's changing and that >> some pieces of data will be relocated. It can work with the current >> memcached version and I have control over all the clients so can impose a >> wrapper around the memcached client api. >> - Option 2 consists in finding a memcached version with a getl/unl >> command (which I read about in the couchbase documentation), it seems a >> natural extension to memcached. I would be acceptable to get just the patch >> and patch the actual version to get just that. >> - Option 3 consists in implementing a rename command within memcached. >> Least preferable obviously since I would have to come up with the code and >> I am blissfully ignorant of the internals of memcached at the moment. On >> other hand it could be a worthy contribution I think since this is not >> something that can be emulated. >> >> >> >> On Saturday, June 16, 2012 7:47:58 PM UTC+2, Nelz wrote: >>> >>> The problem that you are having is that you aren't using memcached as >>> a cache in that case... You are using it as a data-store. >>> >>> I'd suggest using other tools (Membase? Redis?) that fit your >>> requirements, because if you keep trying to use memcached in a >>> use-case for other than it is designed for, you will continually run >>> into problems. >>> >>> - Nelz >>> >>> On Sat, Jun 16, 2012 at 10:17 AM, Matthieu M. >>> <[email protected]> wrote: >>> > >>> > >>> > On Saturday, June 16, 2012 12:08:31 AM UTC+2, Henrik Schröder wrote: >>> >> >>> >> Why don't you just take the cache misses and recreate your data from >>> the >>> >> underlying datastore? If you are using consistent hashing and you >>> change >>> >> your cluster slightly, you lose only a fraction of your cached >>> values. >>> >> >>> >> >>> >> /Henrik >>> >> >>> >> On Fri, Jun 15, 2012 at 7:05 PM, Matthieu M. < >>> [email protected]> >>> >> wrote: >>> >>> >>> >>> I did not find a "rename" operation as I combed the docs, which >>> would >>> >>> have been my savior here, so I would like to know how everyone else >>> handles >>> >>> this situation. >>> >>> >>> > >>> > Unfortunately I am counting on using memcached for highly volatile >>> data with >>> > asynchronous reliable storage (I am thinking thousands to tens of >>> thousands >>> > of writes per seconds on those simple counters), so in case of cache >>> miss I >>> > have to wait for a while (tens of seconds to minutes) before I can >>> reliably >>> > know that all updates were flushed to the data store and restore. >>> > >>> > I just came over an extension to the memcached protocol today (get >>> with >>> > lock: getl and its counterpart unlock: unl) that is experimented with >>> and >>> > could help achieve what I want, unfortunately it might be a while >>> before it >>> > is available in memcached core. >>> > >>> > -- Matthieu >>> >> >
