Yup :) looks like you left a system installed version of memcached running on the other one.
On Mon, 16 Dec 2019, Alireza Sanaee wrote: > Hi, > I did a stupid mistake, the big machine Memcached version seems different. I > guess that is the problem. > > Thanks, > Alireza > > On Mon, Dec 16, 2019 at 2:16 AM Alireza Sanaee <sarsan...@gmail.com> wrote: > Hi, > I'm investigating the Linux load balancer, meanwhile, I'm trying to > understand what is happening in Memcached and just noticed the different > number > of Memcached threads on my machines. Linux doesn't provide immediate access > to generally latency-sensitive threads/processes like Memcached worker > threads, causing HoL and eventually long-tail latency, not a new thing > though. > > But before that, I should know what worker threads I need to consider. You > gave me some pointers to different internal threads of Memcached though. > Even intermittent occurrence of some internal worker threads of > Memcached(rebalancer, crawler or ...) might block some requests (I'm not sure > if > that's the case or not). ULE scheduler sounds suffering from the same flaw. > > The connection dispatcher is there, but I don't have so many connections > creation, I have a good number of clients, and send requests at some rates > until the end of the experiment. I'm using mutilate as my workload generator. > > Sure, I can check whether those are idle or not, I'm actually recording > everything from `/proc/<memcached-pid>/..../stat` so the status is also > available there. It is a true fact that internal threads are IDLE most of the > time and it is some sort of visible in the plot, but I just want to > make sure that all mostly idle ones are just internal threads and not > workers. As you said some times workloads are not loaded evenly making the > worker threads more difficult to distinguish. I think this doesn't matter > now. > > The main issue here is that I have 6 threads on my big machine and 10 threads > on my small machine, while I have the same Memcached configuration for > both machines. I have attached the numbers for the two machines. > > Thanks, > Alireza > > On Sun, Dec 15, 2019 at 3:40 PM dormando <dorma...@rydia.net> wrote: > What're you trying to accomplish? > > Can you include the output of "stats" and "stats settings" on both > machines? > > Dumb question but you've looked at the output of `ps auxH`? If just > using > htop you may not see the threads that're idle. > > TCP connections are pinned to a specific worker thread on connection. > Trivial benchmarks may not load the worker threads evenly, as the > connections are handed to threads evenly via round robin. > > On Sun, 15 Dec 2019, Alireza Sanaee wrote: > > > Hi, > > Thank you for the information, > > > > Sorry for miss using the word there, yes that's all threads. I'm > using the Memcached 1.5.20. I build it myself and then run my > experiments($MEMCACHED -u > > root -p 11211 -m $MAXMEM -c 1024 -t $MEMCACHED_THREADS). And I'm > checking the number of Memcached threads in htop output. It showed me > 10 threads(workers > > included) in one machine and 6 threads(workers included) on the other > one. > > > > To share some more information, I have 200GB of memory for the bigger > machine that creates only 6 threads, and we have only 16GB of > memory for the machine > > that creates 10 threads. I'm just thinking maybe because the smaller > machine has less amount of space, and I'm actually filling in up > to 15GB then I might > > have more work to do and creates more threads. > > > > According to your information, I should expect at least 5 threads > other than the main workers. So 10 threads look OK, but how about > the bigger machine > > which spawns only 6 threads? > > > > I also had difficulties in detecting the worker threads that respond > to GET/SET requests on my results, I have attached two pictures, > one of them shows > > the actual location of each worker on various cores, and the second > one is showing userspace time spent for each worker. Apparently > worker thread number > > 1,2,4 and 5 have spent more time in userspace, so I'm concluding here > that 1,2,4 and 5 are my actual worker threads, and worker 3 and > 6 are just internal > > worker threads of Memcached. Does that make sense to you? > > > > Thanks, > > Alireza > > > > > > On Sun, Dec 15, 2019 at 7:19 AM dormando <dorma...@rydia.net> wrote: > > What version of memcached is on each machine? > > > > memcached doesn't use processes, it's multi-threaded. Different > versions > > may have a different number of background threads. In the > latest version > > there should be at least: > > > > - listener thread (main "process") > > - N worker threads > > - hash table maintenance thread > > - async log thread (for `watch` commands) > > - LRU maintainer thread > > - LRU crawler thread > > - slab rebalancer thread > > > > they're all idle unless they need to do work. LRU maintenance > thread is > > probably the most active, since it executes LRU maintenance > work deferred > > from the worker threads. Older versions have some of these > threads, but > > they were not enabled by default until 1.5.0. > > > > -Dormando > > > > On Sat, 14 Dec 2019, Alireza Sanaee wrote: > > > > > Hello, > > > I'm running Memcached on two different machines with > different specifications. And I specify the number of worker threads = 4 > for both > > machines. However, > > > the number of child processes of the Memcached server is > different on two machines. On one of them, I have 6 Memcached child > processes, and > > on the other > > > server, I have 10 Memcached child processes. I'm curious to > understand how many children processes Memcached is basically > spawning other > > than the worker > > > threads, and for what tasks? > > > > > > I expect the Memcached to spawn only 4 children processes or > a certain number of children processes on two machines, however, > it seems not > > true. > > > > > > Thanks, > > > Alireza > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the > Google Groups "memcached" group. > > > To unsubscribe from this group and stop receiving emails from > it, send an email to memcached+unsubscr...@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/memcached/da7f492f-c12d-4763-86fc-d07311d21c5a%40googlegroups.com. > > > > > > > > > > -- > > > > --- > > You received this message because you are subscribed to a topic > in the Google Groups "memcached" group. > > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/memcached/L05nSQruHRg/unsubscribe. > > To unsubscribe from this group and all its topics, send an > email to memcached+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912141516160.3156%40dskull. > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, > send an email to memcached+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/memcached/CAL%3D0poZ-A0yCr%3DLAAoqcC041bEfC9rthdVZRHCVV-f7xUJ-e4g%40mail.gmail.com. > > > > > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "memcached" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/memcached/L05nSQruHRg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > memcached+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912142337470.3156%40dskull. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/CAL%3D0pobwb6VSGLcPONdvrk4nQJAXzk_mypxiZCYmSSTtheyeVA%40mail.gmail.com. > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912151329110.3156%40dskull.
