+1 to everything dormando just said.
Brian.
--------
http://brian.moonspot.net/
On 4/10/14, 13:53 , dormando wrote:
You really really really really really *must* not put data in memcached
which you can't lose.
Seriously, really don't do it. If you need persistence, try using a redis
instance for the persistent stuff, and use memcached for your cache stuff.
I don't see why you feel like you need to write your own thing, there're a
lot of persistent key/value stores (kyotocabinet/etc?). They have a much
lower request ceiling and don't handle the LRU/cache pattern as well, but
that's why you can use both.
Again, please please don't do it. You are damaging your company. You are a
*danger* to your company.
On Thu, 10 Apr 2014, Slawomir Pryczek wrote:
Hi Dormando, thanks for suggestions, background thread would be nice...
The idea is actually that with 2-3GB i get plenty of evictions of items that
need to be fetched later. And with 16GB i still get evictions,
actually probably i could throw more memory than 16G and it'd only result in
more expired items sitting in the middle of slabs, forever... Now im
going for persistence. Sounds probably crazy, but we're having some data that
we can't loose:
1. statistics, we aggregate writes to DB using memcached (+list
implementation). If these items get evicted we're loosing rows in db. Loosing
data
sometimes isn't a big problem. Eg. we restart memcached once a week so we're
loosing 1 minute of data every week. But if we have evictions we're
loosing data constantly (which we can't have)
2. we drive load balancer using data in memcached for statistics, again, not
nice to loose data often because workers can get incorrect amount of
traffic.
3. we're doing some adserving optimizations, eg. counting per-domain ad
priority, for one domain it takes about 10 seconds to analyze all data and
create list of ads, so can't be done online... we put result of this in
memcached, if we loose too much of this the system will start to serve
suboptimal ads (because it'll need to switch to more general data or much
simpler algorithm that can be done instantly)
Probably would be best to rewrite all this using C or golang, and use memcached
just for caching, but it'd take too much time which we don't have
currently...
I have seen twitter and nk implementations that seem to do what i need, but
they seem old (based on old code), so I prefer to modify code of recent
"official" memcached, to not be stuck with old code or abandonware. Actually
there are many topics about limitations of currrent eviction algo and
option to enable some background thread to do scraping based on statistics of
most filled slabs (with some parameter to specify if it should take
light or aggressive approach) would be nice...
As for the code... is that slab_rebalance_move function in slab.c? It seems a
little difficult to gasp without some DOCs of how things are
working... can you please write a very short description of how this "angry
birds" more workd?
Look at doc/protocol.txt for explanations of the slab move options. the
names are greppable back to the source.
I have quick question about this above... linked is item that's placed on
linked list, but what other flags means, and why 2 last are 2 of them
temporary?
#define ITEM_LINKED 1
#define ITEM_CAS 2
/* temp */
#define ITEM_SLABBED 4
#define ITEM_FETCHED 8
This from slab_rebalance_move seems interesting:
refcount = refcount_incr(&it->refcount);
...
if (refcount == 1) { /* item is unlinked, unused */
...
} else if (refcount == 2) { /* item is linked but not busy */
Is there some docs about refcounts, locks and item states? Basically why item
with refcount 2 is not busy? You're increasing refcount by 1 on
select, then again when reading data? Can refcount ever be higher than 2 (3 in
above case), meaning 2 threads can access same item?
The comment on the same line is explaining exactly what it means.
Unfortunately it's a bit of a crap shoot. I think I wrote a threads
explanation somewhnere (some release notes, or in a file in there, I can't
quite remember offhand). Since scaling the thread code it got a lot more
complicated. You have to be extremely careful under what circumstances you
access items (you must hold an item lock + the refcount must be 2 if you
want to unlink it).
You'll just have to study it a bit, sorry. Grep around to see where the
flags are used.
Thanks.
W dniu czwartek, 10 kwietnia 2014 06:05:30 UTC+2 użytkownik Dormando napisał:
> Hi Guys,
> im running a specific case where i don't want (actually can't have) to
have evicted items (evictions = 0 ideally)... now i have
created some simple
> algo that lock the cache, goes through linked list and evicts items...
it makes some problems, like 10-20ms cache locks on some
cases.
>
> Now im thinking about going through each slab memory (slabs keep a
list of allocated memory regions) ... looking for items, if
expired item is
> found, evict it... this way i can go eg. 10k items or 1MB of memory at
a time + pick slabs with high utilization and run this
"additional" eviction
> only on them... so it'll prevent allocating memory just because
unneded data with short TTL is occupying HEAD of the list.
>
> With this linked list eviction im able to run on 2-3GB of memory...
without it 16GB of memory is exhausted in 1-2h and then memcached
starts to
> kill "good" items (leaving expired ones wasting memory)...
>
> Any comments?
> Thanks.
you're going a bit against the base algorithm. if stuff is falling out of
16GB of memory without ever being utilized again, why is that critical?
Sounds like you're optimizing the numbers instead of actually tuning
anything useful.
That said, you can probably just extend the slab rebalance code. There's
a
hook in there (which I called "Angry birds mode") that drives a slab
rebalance when it'd otherwise run an eviction. That code already safely
walks the slab page for unlocked memory and frees it; you could edit it
slightly to check for expiration and then freelist it into the slab class
instead.
Since it's already a background thread you could further modify it to
just
wake up and walk pages for stuff to evict.
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups "memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
