Thanks for your help on this! So I'm currently bound to this version of memcached (I'm using AWS Elasticache and this is the latest version they support) so will have to make do for now.
It seems before I try decreasing the item expiration time I should experiment with tweaking the LRU sleep time? Would cutting the sleep time say in half be safe or is there a danger there? Also, I found a new memcached term I can't find a lot of info on if you'll indulge me- what exactly are "reflocks"? On Thursday, June 22, 2017 at 9:01:50 PM UTC-4, Dormando wrote: > > Nice. Looks like you actually have new features enabled. > > > STAT crawler_reclaimed 437349953 > > STAT crawler_items_checked 217893516538 > > These stats tell me what I was saying before; you probably have a bunch of > expired stuff in memory, when the crawler wakes up it reclaims a bunch of > it. The spikes you're seeing may be because the crawler is sleeping too > long, or your traffic comes in bursts as well. > > Some scheduling fixes for the LRU crawler went into the latest version, I > think. Should smooth it out a bit. > > you can also do a magic trick: > > telnet to your server, and type "watch evictions" - which will show you > exactly what the objects getting evicted look like. > > > > > ==================================== stats settings > ================================================= > > stats settings > > STAT maxbytes 6787432448 > > STAT launch_time_maxbytes 6787432448 > > STAT maxconns 65000 > > STAT tcpport 11211 > > STAT udpport 11211 > > STAT inter NULL > > STAT verbosity 1 > > STAT oldest 0 > > STAT evictions on > > STAT domain_socket NULL > > STAT umask 700 > > STAT growth_factor 1.15 > > STAT chunk_size 5 > > STAT num_threads 2 > > STAT num_threads_per_udp 2 > > STAT stat_key_prefix : > > STAT detail_enabled no > > STAT reqs_per_event 20 > > STAT cas_enabled yes > > STAT tcp_backlog 1024 > > STAT binding_protocol auto-negotiate > > STAT auth_enabled_sasl no > > STAT item_size_max 1048576 > > STAT config_max 16 > > STAT config_size_max 65536 > > STAT maxconns_fast yes > > STAT hashpower_init 16 > > STAT slab_reassign yes > > STAT slab_automove 0 > > STAT slab_chunk_max 524288 > > STAT lru_crawler yes > > STAT lru_crawler_sleep 100 > > STAT lru_crawler_tocrawl 0 > > STAT tail_repair_time 0 > > STAT flush_enabled yes > > STAT dump_enabled yes > > STAT hash_algorithm murmur3 > > STAT lru_maintainer_thread yes > > STAT hot_lru_pct 32 > > STAT warm_lru_pct 32 > > STAT expirezero_does_not_evict no > > STAT idle_timeout 0 > > STAT watcher_logbuf_size 262144 > > STAT worker_logbuf_size 65536 > > STAT track_sizes no > > > > > > > > > > On Thursday, June 22, 2017 at 7:53:56 PM UTC-4, Dormando wrote: > > Can you paste the full output from the 'stats' command against the > server? > > Also: "stats settings" would be great. > > > > Also; do you know anything about your data? what the TTL's are, > for > > instance? It's possible a bunch of stuff expires, which can get > reclaimed > > quickly by some processes depending on your start arguments. > > > > On Thu, 22 Jun 2017, red 888 wrote: > > > > > Oh sorry! Version 1.4.34. > > > > > > On Thursday, June 22, 2017 at 4:52:18 PM UTC-4, Dormando wrote: > > > Think I've asked this a few times; what version are you > running? > > > > > > On Thu, 22 Jun 2017, red 888 wrote: > > > > > > > So when I look at my stats I see no cmd_flush commands > are being run. cmd_flush > > is 0 > > > and stays at 0. > > > > > > > > On Thursday, June 22, 2017 at 12:44:28 PM UTC-4, > Dormando wrote: > > > > Looks like a weird access pattern. you're filling > memory, evicting a bit, > > > > then losing all of it at once. Is your "cmd_flush" > counter increasing? > > > > > > > > On Thu, 22 Jun 2017, red 888 wrote: > > > > > > > > > If I look at the slab stats over time I see the > % of chunks_used > > (relative to > > > > total_chunks) spike up (some slab classes hit 45% > some hit 75%) then > > spike > > > > > down and at the same time they are spiking down > evictions spike up. > > This > > > happens about > > > > every 2-4 minutes. > > > > > > > > > > I've attached a screen shot of this pattern in a > time series (raw stats > > not > > > > phpmemchacheadmin). In the graphs I'm filtering > for just one slab class > > but I > > > > > see this for all the most active slabs ( > roughly 5 active slab > > classes). > > > > > Inline image 2 > > > > > > > > > > On Jun 21, 2017 2:27 PM, "dormando" < > [email protected]> wrote: > > > > > Is the evictions counter increasing all > the time? It might only > > be > > > > > completely full sometimes, which would > cause evictions (like > > during > > > peak). > > > > > > > > > > if used is less than total but evictions > is going up, either you > > have an > > > > > old broken version of memcached or that > phpmemcachedadmin thing > > is wrong, > > > > > and you should just look at the output of > the "stats items" and > > "stats > > > > > slabs" commands. > > > > > > > > > > On Wed, 21 Jun 2017, red 888 wrote: > > > > > > > > > > > Here is a snap shot of one of my slab > class's stats (from > > > PHPMemcachedAdmin): > > > > > > > > > > > > [Capture.PNG] > > > > > > > > > > > > > > > > > > So what I'm confused about is why, with > my used_chunks so low, > > am I > > > getting > > > > evictions for this slab class? > > > > > > > > > > > > > > > > > > used_chunks are chunks with items that > have not expired yet and > > > total_chunks is > > > > all allocated chunks including chunks with items > that > > > > > _have_ expired > > > > > > correct? > > > > > > > > > > > > > > > > > > If thats the case it means if 25% of the > allocated chunks are > > "used" > > > and 75% of > > > > the allocated chunks have expired. So it should > have plenty > > > > > of chunks to > > > > > > reclaim before having to evict I would > think. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > --- > > > > > > You received this message because you > are subscribed to the > > Google > > > Groups > > > > "memcached" group. > > > > > > To unsubscribe from this group and stop > receiving emails from > > it, send > > > an email > > > > to [email protected]. > > > > > > For more options, visit > https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > --- > > > > > You received this message because you are > subscribed to a topic > > in the > > > Google > > > > Groups "memcached" group. > > > > > To unsubscribe from this topic, visit > > > > > https://groups.google.com/d/topic/memcached/VXm-hnGzqXY/unsubscribe. > > > > > To unsubscribe from this group and all its > topics, send an email > > to > > > > [email protected]. > > > > > For more options, visit > https://groups.google.com/d/optout. > > > > > > > > > > -- > > > > > > > > > > --- > > > > > You received this message because you are > subscribed to the Google > > Groups > > > "memcached" > > > > group. > > > > > To unsubscribe from this group and stop > receiving emails from it, send > > an email > > > to > > > > [email protected]. > > > > > For more options, visit > https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > -- > > > > > > > > --- > > > > You received this message because you are subscribed to > the Google Groups > > "memcached" > > > group. > > > > To unsubscribe from this group and stop receiving emails > from it, send an email > > to > > > > [email protected]. > > > > For more options, visit > https://groups.google.com/d/optout. > > > > > > > > > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the > Google Groups "memcached" > > group. > > > To unsubscribe from this group and stop receiving emails from > it, send an email to > > > [email protected]. > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to > > [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > > > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
