I run every minute stats, stats items and stats slabs. the only commands executed are remove, incr, add, get, set and cas.
I'm running now with 6 threads per instance with 3 per server and haven't had the issue again, not that this change fixed it. I'll definitely update. On Aug 7, 2014 6:13 PM, "dormando" <[email protected]> wrote: > Please upgrade. If you have problems with the latest version we can look > into it more. > > You can also look at command counters for odd commands being given: make > sure nobody's running flushes, or "stats sizes", or "stats cachedump" > since those can cause CPU spikes and hangs. > > With 1.4.20 you can use "stats conns" to see what the connections are > doing during the cpu spike. > > On Thu, 7 Aug 2014, Claudio Santana wrote: > > > Forgot to say I'm running version 1.4.13 libevent 2.0.16-stable > > > > > > > > On Thu, Aug 7, 2014 at 6:08 PM, Claudio Santana < > [email protected]> wrote: > > Sorry for the late response. > > > > My CPU utilization normally is min 2.5% to 6.5% max. > > > > So it's interesting you ask this. The reason why I submitted the 1st > question is because I've experienced some random CPU > > utilization spikes. From this about 6% CPU utilization all of the sudden > it spikes to 100% and I can see the offending > > process is one of the Memcached instances. Sadly this CPU spike is > accompanied by all requests timing out causing the > > whole system to become unusable. > > > > I collect minute by minute stats of all these memcached instances and > according to my stats this issue happens within 2 > > minutes. I can see in the number of commands there's no increase in > number of commands being issued right before the CPU > > spike nor increase in the number of bytes in/out. > > > > Does anybody have any ideas of what could be going on? > > > > I have all Memcached stats collected by minute in Graphite, I can > provide other stats that could help explain this issue > > if necessary. > > > > > > On Mon, Aug 4, 2014 at 9:36 PM, dormando <[email protected]> wrote: > > You could run one instance with one thread and serve all of that > just > > fine. have you actually looked at graphs of the CPU usage of the > host? > > memcached should be practically idle with load that low. > > > > One with -t 6 or -t 8 would do it just fine. > > > > On Mon, 4 Aug 2014, Claudio Santana wrote: > > > > > Dormando, thanks for the quick response. Sorry for the > confusion, I don't have exact metrics per second but > > per minute 1.12 > > > million sets and 1.8 million gets which translates to 18,666 > sets per minute and 30,000 gets per second. > > > > > > These stats are per Memcached instance which I currently run 3 > on each server. > > > > > > Claudio. > > > > > > > > > On Mon, Aug 4, 2014 at 6:22 PM, dormando <[email protected]> > wrote: > > > On Mon, 4 Aug 2014, Claudio Santana wrote: > > > > > > > I have this Memcached cluster where 3 instances of > Memcached run in a single server. These servers > > have 24 cores, > > > each instance > > > > is configured to have 8 threads each. Each individual > instance serves have about 5000G gets/sets a > > day and about > > > 3k current > > > > connections. > > > > > > I don't know what "5000G gets/sets a day" translates to in > per-second (nor > > > what the G-unit even is?), can you define this? > > > > > > > What would be better? consolidate these 3 instances to a > single instance per server with 24 threads? I've > > read in a few > > > articles > > > > that Memcached's performance starts suffering with more than > 4-6 threads per instance, is this generally > > true? > > > > > > > > How about keeping the 3 instances per server and decreasing > the number of threads to say 4 or 6? or > > creating 4 instances > > > in the > > > > same servers instead of 3 and decreasing the number of threads > per instance to 6 so there is one thread > > per core. > > > > > > > > Is there a guide you could recommend to configure the right > number of threads and strategies to get the > > most out of a > > > Memcached > > > > server/instance? > > > > > > > > Thanks, > > > > Claudio > > > > > > > > -- > > > > > > > > --- > > > > You received this message because you are subscribed to the > Google Groups "memcached" group. > > > > To unsubscribe from this group and stop receiving emails from > it, send an email to > > > [email protected]. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the > Google Groups "memcached" group. > > > To unsubscribe from this group and stop receiving emails from > it, send an email to > > [email protected]. > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the > Google Groups "memcached" group. > > > To unsubscribe from this group and stop receiving emails from > it, send an email to > > [email protected]. > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, > send an email to > > [email protected]. > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
